Newshot

Jsonbit – Reduce token usage when AI reads logs

Original text(via HN AI)

jsonbit

题图
题图

JsonBit compacts nested JSON before it reaches your LLM. Send a compact digest in one prompt, or let your agent query the original data locally via SQL/MCP and read only the results it needs.


        pip install jsonbit
jsonbit digest path/to/your-file.json
      

Pass the output to your LLM in place of the raw JSON. (Rust cargo, Node.js npm/npx, and standalone binaries are listed under Installation & Setup.)

Example output from jsonbit digest:


        # orders[4](orderId,customer_name,customer_tier,status,total)
@stats: total[sum=619.00,avg=154.75,min=95.50,max=201.00]
ORD-101,Alice,pro,shipped,171.75
ORD-102,Bob,starter,pending,201
ORD-103,Cara,pro,shipped,95.5
ORD-104,Dan,pro,shipped,150.75
# orders_items[6] -> orders (sku,qty,price)
@stats: qty[sum=15,avg=2.5,min=1(n=1),max=4] price[sum=246.25,avg=41.04,min=10.50,max=74.50]
0:USB-C,2,10.5|MOUSE,3,50.25
1:MOUSE,4,50.25
2:HUB-4,1,74.5|USB-C,2,10.5
3:MOUSE,3,50.25
      
WorkflowCommand / MCP ToolBest ForFidelity & Guarantees
1-Turn Compactionjsonbit digest / olap_digestSingle-prompt LLM calls & holistic summaries where the model needs every row.Compact LLM text view containing all rows, escaped delimiters, hoisted child tables, and precomputed stats (rounded averages are approximate) .
Agentic Local Queryingjsonbit card + sql / olap_load + olap_sqlCoding agents (OpenCode, Claude Code, Cursor) querying medium/large .json or .jsonl files.Emits a compact schema and statistics summary (independent of row count); runs exact read-only SQL & BitVec64 filters locally in microseconds.
Binary Storagejsonbit pack / jsonbit rehydrateArchiving or caching normalized .jsonbit snapshots on disk ( -80.6% vs. minified JSON on W1 – W10 ).Lossless round-trip verified across all 33 benchmark/adversarial datasets and unit tests (preserving nulls, colliding keys, and u64 > i64::MAX ).

Table of Contents

  • Installation & Setup
  • Agent Integrations (OpenCode, Claude Code, Codex, Cursor)
  • CLI Reference
  • Architecture

Installation & Setup

Install the pre-built native jsonbit binary using your preferred package manager (no Rust compiler required for Python or Node.js):

Python (pip / uv) — Includes native CLI + Python SDK


        pip install jsonbit
      

Rust (cargo)


        cargo install jsonbit
      

Node.js (npm / npx) — Includes native CLI + TypeScript SDK


        npm install -g @jsonbit/core
# or run directly without global install:
npx @jsonbit/core --help
      

Pre-Built Binaries & Building from Source

Standalone binaries for Linux (x64, arm64), macOS (arm64, x64), and Windows (x64) are attached to every GitHub Release.

To build from source (requires Rust 1.75+):


        git clone https://github.com/prathikmurugan/jsonbit.git
cd jsonbit
cargo install --path .
      

Verify installation:


        jsonbit --version
      

Agent Integrations

jsonbit includes a built-in stdio JSON-RPC 2.0 MCP server (jsonbit mcp) with zero external runtime dependencies. It exposes five tools: olap_load, olap_digest, olap_ask, olap_sql, and olap_rehydrate.

OpenCode (opencode.json)

Add the following to opencode.json in your project root (or globally in ~/.config/opencode/opencode.json):


        {
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "jsonbit": {
      "type": "local",
      "command": ["jsonbit", "mcp"],
      "enabled": true
    }
  }
}
      

Claude Code (.mcp.json or CLI)

Register via the Claude Code CLI:


        claude mcp add jsonbit -- jsonbit mcp
      

Or include in .mcp.json:


        {
  "mcpServers": {
    "jsonbit": {
      "command": "jsonbit",
      "args": ["mcp"]
    }
  }
}
      

Cursor & Windsurf

Add the .mcp.json configuration above to .cursor/mcp.json or ~/.codeium/windsurf/mcp_config.json.

OpenAI Codex CLI (AGENTS.md)

Codex and terminal agents automatically read AGENTS.md in the repository root to invoke jsonbit via CLI or MCP when inspecting .json, .jsonl, or .jsonbit files.

CLI Reference

CommandArgumentsDescription
jsonbit card<file.json│.jsonl│.jsonbit>Prints the compact 3NF schema and statistics summary (independent of row count) plus storage compression stats.
jsonbit digest<file.json│.jsonl│.jsonbit>Prints the 1-Turn Factorized 3NF Holistic Digest (all rows, hoisted child tables, and precomputed summary stats).
jsonbit ask<file> "<question>"Evaluates a question against the bitmap index and prints the verified context slice.
jsonbit answer<file> "<question>"Evaluates a question and prints only the scalar answer value.
jsonbit sql<file> "<SELECT ...>"Executes a read-only SQL query against the 3NF columnar store ( WHERE , IN , BETWEEN , LIKE , GROUP BY , ORDER BY , LIMIT , OFFSET , CSR JOIN ).
jsonbit pack<input.json> <output.jsonbit>Serializes the bit-packed 3NF tables into a binary .jsonbit snapshot.
jsonbit rehydrate<file.json│.jsonl│.jsonbit>Reconstructs and prints the original nested JSON document.
jsonbit mcp(none)Starts the concurrent stdio JSON-RPC 2.0 MCP server ( olap_load , olap_digest , olap_ask , olap_sql , olap_rehydrate ).

Python (pip install jsonbit)


        from jsonbit import JsonBit

with JsonBit("benchmarks/data/github-repos.json") as olap:
    card = olap.card()
    ans = olap.answer("How many repositories have more than 150000 stars and more than 30000 forks?")
    rows = olap.sql("SELECT defaultBranch, COUNT(*) AS cnt FROM records GROUP BY defaultBranch ORDER BY cnt DESC")
    olap.pack("repos.jsonbit")
      

TypeScript / Node.js (@jsonbit/core)


        import { JsonBit, rehydrateFrom3NF } from '@jsonbit/core'

const engine = JsonBit.from(jsonData)
const card = engine.getDataCard().text
const rows = engine.sql(`SELECT orderId, customer.name, total FROM orders WHERE total > 500`)
const original = rehydrateFrom3NF(engine)
engine.close()
      

Rust Library (cargo add jsonbit)


        use jsonbit::{execute_sql, JsonBitEngine};

let engine = JsonBitEngine::from_json(&serde_json::from_str(raw_json)?);
let card = engine.data_card();
let rows = execute_sql(&engine, "SELECT department, COUNT(*) AS cnt FROM employees GROUP BY department")?;
let restored = engine.rehydrate();
      

Architecture


        flowchart TD
    Input["Nested JSON / JSONL Input"] --> Normalizer["3NF Normalizer & CSR Indexer (rust/normalizer.rs)"]
    Normalizer --> ColStore["Bit-Packed Columnar Store (rust/column.rs)"]
    ColStore --> Bitmap["BitVec64 Bitmap Engine (rust/bitmap.rs)"]
    Bitmap --> Card["OLAP Data Card (rust/engine.rs)"]
    Bitmap --> SqlExec["SQL & Predicate Executor (rust/query_exec.rs)"]
    ColStore --> Snapshot[".jsonbit Binary Snapshot (rust/snapshot.rs)"]
    ColStore --> Rehydrate["Lossless 3NF Rehydrator (rust/normalizer.rs)"]
      

Storage Layout (rust/column.rs & rust/normalizer.rs)

  • ForIntColumn (Frame-of-Reference Integers): Computes [min_val, max_val] per column and packs unsigned deltas val - min_val into ceil(log2(max_val - min_val + 1)) bits per row inside a Vec<u64> buffer. Column headers store sum, min, max, and non_null_count for O(1) unfiltered aggregations.
  • DictStringColumn (Dictionary + Inverted Bitmaps): Interns unique strings into a string arena, bit-packs dictionary IDs into ceil(log2(|D|)) bits per row, and builds a 64-bit word-aligned BitVec64 inverted index per category (|D| <= 64) for O(1) equality lookups and POPCNT frequency histograms.
  • BoolBitmapColumn (1-Bit Booleans): Packs 64 boolean values into each u64 word alongside a 1-bit validity mask.
  • CSR 1-to-Many Join Index: Child arrays (e.g., orders[].items[]) are normalized into child tables with a contiguous csr_offsets: Vec<u32> array mapping each parent row i to its child slice csr_offsets[i]..csr_offsets[i + 1].

Benchmarks

All benchmarks execute directly against the compiled Pure-Rust jsonbit binary (target/release/jsonbit).

1. Live gemini-3.8-flash Evaluation Across 10 Real-World Workloads (27 Questions)

Evaluates Raw JSON, Minified JSON, YAML, TOON (@toon-format/toon), and jsonbit across 10 production workloads (W1–W10), including flat tables, 1-to-Many e-commerce orders, sparse .jsonl OpenTelemetry logs with free-text messages, singleton Kubernetes configs, IAM primitive arrays, wide 30-column tables, and multi-byte UTF-8 financial ledgers. Across the 27 questions, single-prompt jsonbit digest uses 57.4% fewer estimated tokens than Minified JSON (101,302 vs. 237,891) with 26/27 vs. 13/27 accuracy:

Format / ToolModeLive gemini-3.8-flash AccuracyEst. Tokens ( o200k_base )Reduction vs. Minified JSONReduction vs. Raw JSON
Raw JSON (Pretty)1-Turn Full Text55.6% (15/27)375,713+57.9%—
Minified JSON1-Turn Full Text48.1% (13/27)237,891—-36.7%
YAML1-Turn Full Text59.3% (16/27)293,690+23.5%-21.8%
TOON v4 ( @toon-format/toon )1-Turn Full Text59.3% (16/27)169,627-28.7%-54.9%
jsonbit Holistic Digest ( olap_digest )1-Turn (All Rows + Summary Stats)96.3% (26/27)101,302-57.4%-73.0%

Agentic local-query comparison (separate workflow — schema summary + Rust SQL execution, not a full-row text encoding):

ToolModeLive gemini-3.8-flash AccuracyEst. Tokens ( o200k_base )Reduction vs. Minified JSON
jsonbit Schema Card + SQL ( olap_load + olap_sql )Agentic Schema Summary + Local Rust SQL100.0% (27/27)11,729-95.1%
Methodology note: Token counts above are gpt-tokenizer (o200k_base) estimates for reproducible cross-format comparison; the jsonbit SQL path total includes the schema summary prompt plus generated SQL query tokens.

Single-Prompt Summary Accuracy (W3, W4, W5, W10)

For qualitative tasks where the LLM must read all records in a single turn and write a multi-fact executive summary (naming all departments, products, microservice failure messages, and international counterparties across W3, W4, W5, and W10):

Holistic Synthesis TaskTOON v4 Facts Preservedjsonbit digest Facts PreservedTOON Tokensjsonbit digest TokensToken Savings vs. TOON
H1 — HR Directory Executive Summary8 / 99 / 9 (100%)4,8814,372-10.4%
H2 — E-Commerce Catalog & Orders Summary8 / 88 / 8 (100%)15,4056,933-55.0%
H3 — Production Incident Log Triage Report9 / 99 / 9 (100%)13,6654,169-69.5%
H4 — International Settlement Ledger Summary2 / 66 / 6 (100%)3,7812,403-36.4%
Total Across All Qualitative Summaries27 / 32 (84.4%)32 / 32 (100.0%)37,73217,877-52.6%

2. Token Scaling Curve (4 $\rightarrow$ 25,000 Nested Orders)

While linear text encodings (Raw JSON, Minified JSON, YAML, TOON) scale at $O(N)$ with row count and exceed LLM context windows at scale, jsonbit maintains an $O(1)$ prompt footprint (~67–473 tokens):

Dataset Scale (Nested Orders)Raw JSON (Tokens)YAML (Tokens)TOON v4 (Tokens)jsonbit Rust Card (Tokens)Rust Ingest Time
4 rows (Tiny API Payload)16612271670.04 ms
50 rows (~150 line items)7,7805,8014,7834450.51 ms
500 rows (~1,500 line items)78,11558,24447,9834473.99 ms
5,000 rows (~15,000 line items)783,443366,356293,97247055.56 ms
25,000 rows (~75,000 line items)3,917,3661,831,8431,469,911471315.13 ms
Note: For 5,000- and 25,000-row payloads (>2 MB), YAML and TOON token counts are ratio-extrapolated from the 500-row exact encoding; jsonbit card tokens and Rust ingest times are measured directly on the full dataset.

3. Memory Footprint & Query Speed (Pure-Rust jsonbit vs. jq CLI)

WorkloadCategoryMinified JSONRust Column-Store (RAM)Pretty JSON (Tok)TOON (100% Rows)jsonbit digest (100% Rows)jsonbit card ($O(1)$)3NF Lossless
W1-tiny-api-responseTiny Payload (<500B)0.3 KB0.1 KB1667167 (-5.6%)67PASS
W2-singleton-k8s-configSingleton Config Tree0.3 KB0.2 KB167131110 (-16.0%)110PASS
W3-hr-payroll-directoryTabular (200 rows)28.4 KB4.5 KB14,0644,8474,327 (-10.7%)270PASS
W4-ecommerce-orders-1toN1-to-Many (150 orders)48.3 KB4.5 KB25,46615,3726,892 (-55.2%)470PASS
W5-otel-production-logsSparse .jsonl + Free-Text44.4 KB4.7 KB17,43913,6324,132 (-69.7%)224PASS
W6-github-repos-metricsHigh-Card + Ranges28.6 KB3.4 KB14,8295,3644,414 (-17.7%)307PASS
W7-stripe-dispute-webhooksFinancial Webhooks28.9 KB3.1 KB14,3334,1044,341 (+5.8%)248PASS
W8-iam-primitive-arraysPrimitive Arrays14.4 KB1.6 KB7,1854,8101,257 (-73.9%)162PASS
W9-wide-enterprise-crmWide Table (30 cols)68.6 KB5.7 KB40,7799,8069,549 (-2.6%)1,134PASS
W10-unicode-financial-ledgerUTF-8, Nulls & Zeroes18.4 KB2.9 KB9,5503,7502,364 (-37.0%)241PASS
Total (10 Workloads)All Categories280.6 KB30.6 KB (-89.1%)143,97861,88737,453 (-39.5%)3,233 (-94.8%)10/10 PASS
Workload breakdown: On tiny payloads (W1), singleton configs (W2), financial webhooks with heavy quoting (W7, +5.8%), and wide flat 30-column tables (W9, -2.6%), 1-turn digest is close to TOON. The largest reductions come from nested 1-to-Many hierarchies (W4, -55.2%), sparse .jsonl logs (W5, -69.7%), and primitive arrays (W8, -73.8%), where 3NF hoisting and prefix/dictionary factoring eliminate repeated structure.
  • Storage Accounting (W1–W10): In-memory bit-packed column store: 280.6 KB (Minified JSON) $\rightarrow$ 30.6 KB (-89.1%) Serialized on-disk .jsonbit snapshots (jsonbit pack): 280.6 KB (Minified JSON) / 427.7 KB (raw files) $\rightarrow$ 54.3 KB (-80.6% vs. Minified JSON, -87.3% vs. raw files)
  • In-memory bit-packed column store: 280.6 KB (Minified JSON) $\rightarrow$ 30.6 KB (-89.1%)
  • Serialized on-disk .jsonbit snapshots (jsonbit pack): 280.6 KB (Minified JSON) / 427.7 KB (raw files) $\rightarrow$ 54.3 KB (-80.6% vs. Minified JSON, -87.3% vs. raw files)
  • Query Latency (Warm In-Memory Rust Engine vs. Cold jq CLI Subprocess): jq CLI (stedolan/jq, cold subprocess including startup & JSON parsing): 6,545.3 µs average per query jsonbit Pure-Rust SQL (execute_sql, warm in-memory engine): 6.8 µs average per query jsonbit Pure-Rust Bitmap Slice (query_slice, warm in-memory engine): 19.9 µs average per query
  • jq CLI (stedolan/jq, cold subprocess including startup & JSON parsing): 6,545.3 µs average per query
  • jsonbit Pure-Rust SQL (execute_sql, warm in-memory engine): 6.8 µs average per query
  • jsonbit Pure-Rust Bitmap Slice (query_slice, warm in-memory engine): 19.9 µs average per query
  • Standard 8-Dataset Benchmark Suite: In-Memory Column-Store Compression: 1,178,283 bytes (Minified JSON) $\rightarrow$ 419,661 bytes (-64.4%), 8/8 (100%) lossless 3NF rehydration. Schema Card Size vs. Full Dataset Text: 334,046 tokens (TOON v4 full text) $\rightarrow$ 2,536 tokens (-99.24%, schema & stats summary only). 239-Question Evaluation Accuracy: 99.16% (237/239) strict match at 8.0 µs average warm query latency.
  • In-Memory Column-Store Compression: 1,178,283 bytes (Minified JSON) $\rightarrow$ 419,661 bytes (-64.4%), 8/8 (100%) lossless 3NF rehydration.
  • Schema Card Size vs. Full Dataset Text: 334,046 tokens (TOON v4 full text) $\rightarrow$ 2,536 tokens (-99.24%, schema & stats summary only).
  • 239-Question Evaluation Accuracy: 99.16% (237/239) strict match at 8.0 µs average warm query latency.

4. Autonomous Coding Agent Head-to-Head (OpenCode v1.19.5)

Evaluated in isolated directories with zero prompt hints and zero AGENTS.md/CLAUDE.md files (opencode/big-pickle) across both standard (bash-enabled) and sandboxed (bash: "deny") agent environments:

Agent Modejsonbit MCP ( opencode.json only)TOON v4 ( toon_encode / CLI)Raw JSON
Standard Agent ( 6 Scenarios, 18 Runs)6/6 ( 88.1s ) · 0 scripts6/6 ( 142.8s ) · awk /Python parsers6/6 ( 93.7s ) · Python scripts
Sandboxed Read-Only ( bash: "deny" , 8 Scenarios)8/8 ( 77.7s )7/8 ( 333.8s ) · failed 500 -order sum7/8 ( 345.4s ) · failed 500 -order sum

License

MIT License.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗