DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?

A benchmark for distributed-system diagnosis

DistDebug-Bench

When a distributed system fails, can your agent find the code that caused it?

DistDebug-Bench gives agents 120 real failures from 11 production distributed systems. Failures surface far from their cause, and reports, logs, and stack traces show only part of what happened.

Leaderboard

The benchmark

Find the cause.Explain it in code.

Each incident is a real, closed bug from a production system’s issue tracker, most often a logic, concurrency, or error-handling fault. The agent gets the report, its logs or stack traces, and the repository before the fix, and must name the code blocks that caused the failure. Most coding benchmarks grade the patch; DistDebug-Bench scores the diagnosis itself, against human-labeled causal code.

11 systems, 120 incidents

Why the incidents are hard

68%of reports never name the causal method

63%have no usable stack trace from inside the project

46%need more than one code block to explain

5 linesis the median labeled block, so answers must be precise

Where agents fall short.

Block coverage is the share of the labeled root-cause code blocks a diagnosis reaches; block precision is the share of the code it names that is part of the cause.

The localization gap

Finding the file is only the start: across these configurations, coverage falls by 13–22 percentage points when agents must identify the causal code blocks.

• Mini-SWE-Agent: GPT-6 AstraMini-SWE-Agent 86.1 72.6 13.5 pp gap

Mini-SWE-Agent: GPT-6 AstraMini-SWE-Agent

• Mini-SWE-Agent: Claude Opus 5.5Mini-SWE-Agent 84.5 71.8 12.7 pp gap

Mini-SWE-Agent: Claude Opus 5.5Mini-SWE-Agent

• Mini-SWE-Agent: GPT-5.5Mini-SWE-Agent 86.2 70.0 16.2 pp gap

Mini-SWE-Agent: GPT-5.5Mini-SWE-Agent

• Mini-SWE-Agent: GPT-5.4Mini-SWE-Agent 81.0 66.6 14.4 pp gap

Mini-SWE-Agent: GPT-5.4Mini-SWE-Agent

• Mini-SWE-Agent: 5.3-CodexMini-SWE-Agent 79.9 63.2 16.7 pp gap

Mini-SWE-Agent: 5.3-CodexMini-SWE-Agent

• Mini-SWE-Agent: MiMo v2.5 ProMini-SWE-Agent 80.5 62.0 18.5 pp gap

Mini-SWE-Agent: MiMo v2.5 ProMini-SWE-Agent

• Mini-SWE-Agent: Kimi K2.6Mini-SWE-Agent 79.2 62.0 17.2 pp gap

Mini-SWE-Agent: Kimi K2.6Mini-SWE-Agent

• Mini-SWE-Agent: MiMo v2.5Mini-SWE-Agent 79.7 61.1 18.6 pp gap

Mini-SWE-Agent: MiMo v2.5Mini-SWE-Agent

• Mini-SWE-Agent: 5.2-CodexMini-SWE-Agent 76.7 59.5 17.2 pp gap

Mini-SWE-Agent: 5.2-CodexMini-SWE-Agent

• Mini-SWE-Agent: DeepSeek V4 FlashMini-SWE-Agent 77.4 59.2 18.2 pp gap

Mini-SWE-Agent: DeepSeek V4 FlashMini-SWE-Agent

• Mini-SWE-Agent: HY3Mini-SWE-Agent 74.1 54.8 19.3 pp gap

Mini-SWE-Agent: HY3Mini-SWE-Agent

• Mini-SWE-Agent: Gemini 3.5Mini-SWE-Agent 67.7 50.8 16.9 pp gap

Mini-SWE-Agent: Gemini 3.5Mini-SWE-Agent

• Mini-SWE-Agent: o4-miniMini-SWE-Agent 64.2 42.1 22.1 pp gap

Mini-SWE-Agent: o4-miniMini-SWE-Agent

• Codex CLI: 5.2-CodexCodex CLI 65.7 49.9 15.8 pp gap

Codex CLI: 5.2-CodexCodex CLI

• Codex CLI: o4-miniCodex CLI 56.5 35.9 20.6 pp gap

Codex CLI: o4-miniCodex CLI

• OpenCode: 5.2-CodexOpenCode 69.4 51.4 18.0 pp gap

OpenCode: 5.2-CodexOpenCode

• OpenCode: o4-miniOpenCode 55.0 35.1 19.9 pp gap

OpenCode: o4-miniOpenCode

Report evidence

Without a pointer to the code, coverage drops sharply.

Block coverage

• Causal method namedNo internal stack trace · n=16 94.8%

Causal method namedNo internal stack trace · n=16

• No causal code namedNo internal stack trace · n=42 65.6%

No causal code namedNo internal stack trace · n=42

• No causal code namedCross-component crash site · n=7 61.9%

No causal code namedCross-component crash site · n=7

Consistency across runs

Reports without direct code clues show more run-to-run variation.

4.3× the run-to-run variation when the report names no causal code

Variation in block coverage (SD, pp)

• Causal method named · n=16 1.5 pp

Causal method named · n=16

• No causal code named · n=42 6.4 pp

No causal code named · n=42

Incidents crossing the flaky threshold

• Causal method named · n=16 6.2% (1/16)

Causal method named · n=16

• No causal code named · n=42 16.7% (7/42)

No causal code named · n=42

Means average three runs within each incident and weight the 120 incidents equally; whiskers are 95% incident-bootstrap confidence intervals.

Use DistDebug-Bench

Try one incident.Submit the full run.

Try one incident

Run Mini-SWE-Agent on ZooKeeper-2845 with the paper’s protocol. From a benchmark checkout, the runner prepares an anonymized workspace at the buggy ZooKeeper revision, runs the agent in a sandbox, and the scorer applies the paper’s evaluator. Python 3.10+, Linux with bubblewrap, and a configured model API key are required.

        python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[agent]"

# Set OPENAI_API_KEY in your environment.
python -m distdebug_bench.run \
    --config configs/mini-swe-agent/gpt-5.4.yaml \
    --bug zookeeper-2845 --runs 1
python -m distdebug_bench.score \
    results/mini-swe-agent-gpt-5.4 --attempts 1
      

configs/mini-swe-agent/ has a config for each model in the paper. Runs call your configured model provider.

Each attempt writes trajectory.json, diagnosis.json, and diagnosis_scores.json. To evaluate another agent, have it write the first two.

Submit full results

Run your agent on all 120 incidents, three attempts each, and send its diagnoses and trajectories. We rescore every submission with the official evaluator before listing it on the leaderboard.

• 01 / RUNAll 120 incidents, three independent attempts each, with only the report, its evidence, and the pre-fix repository. Any agent or harness.

• 02 / SUBMITA diagnosis.json and the full trajectory for every attempt, plus the model, harness, settings, and token use or cost.

• 03 / SCOREThe paper’s protocol: the first three schema-valid attempts per incident count, and the leakage guard discards flagged attempts.

The guide covers the local leakage check, packaging a run, and the diagnosis.json schema.

Cite

DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems? Jiang et al., NeurIPS 2026. Paper ↗ · GitHub ↗ · Dataset ↗ · Traces ↗

        @inproceedings{jiang2026distdebugbench,
  title = {{DistDebug-Bench}: Can {LLM} Agents
           Diagnose Bugs in Distributed Systems?},
  author = {Jiang, Yuxuan and Liu, Chenglin and
    Rose, John and Miller, Mason and
    Thavamani, Ruthesh and Wu, Juntao and
    Huang, Yinwei and Xu, Jiahao and Huang, Peng},
  booktitle = {Advances in Neural Information
    Processing Systems (NeurIPS)},
  year = {2026}
}