prepass

Agent Retrieval Bench · trace2code · n=101 · Recall@5

First of eight, on someone else's benchmark.

A published benchmark with 287 hand-reviewed samples and a leaderboard for seven retrievers. Given the output of a failing test, find the source at fault. prepass ranks the benchmark's own corpus snapshots and is scored against its published answer key — so neither the questions nor the answers are ours.

Recall@20 reaches 0.941 and MRR 0.574, against 0.165 for the 8-billion-parameter model. Median query time on their corpus: 16ms.

The same benchmark asks two other questions, and prepass is mid-pack on both. A page that showed you only trace2code would be selling you something.

The pattern is not mysterious. A stack trace is a dense list of exact identifiers printed verbatim — the best possible input for a lexical ranker. "Which tests should change with this code?" requires knowing what a test covers, which no amount of word overlap tells you.

Speed

One number would be the wrong shape for this claim.

Latency is dominated by I/O that scales with the candidate pool, so here is all of it — measured end to end across 27 real repositories.

Worst case measured: 221 ms on Django at the 5,000-file discovery cap. Benchmark tables quote 106 ms, which is the SWE-bench median over a 1,867-file pool — larger than most projects, and not what you will see.

Codex desktop, asked a symptom-shaped question about a real iOS codebase. Nothing in the prompt mentions prepass. The agent reached for it on its own, and said why:

That is the whole design in one screenshot: orient first, then narrow — instead of opening with a repository-wide grep.

The raw per-instance results ship in the repository — every candidate list, every rank, every miss. You can recompute the percentages from them without running anything. If you want to reproduce the runs themselves:

# Agent Retrieval Bench — their corpus, their published answer key
ARB_ROOT=<their release> node bench/arb.mjs

# SWE-bench Lite — 300 real GitHub issues
node bench/swebench.mjs --repos <clones> --data swebench-lite.json

Both harnesses exit non-zero rather than print a number they cannot stand behind: zero scored instances is fatal, and so is a run where more instances failed to check out than succeeded. That rule exists because earlier versions did the opposite.

What went wrong

Three numbers on this page were wrong first.

Every measurement error in this project was a definition error — counting one thing and describing it as another. None would have been caught by running the test again, because they were perfectly reproducible and perfectly wrong.

• The ARB answer key. I first derived the correct files myself from the sample records, using a field that looked right. It was a subset of the real answer and wrong on 22 of 287 samples, which inflated recall. Those numbers were thrown away. The figures above read the benchmark's own published per-sample results.
• The latency claim. This tool was described as "110ms" for several days. That was the SWE-bench median — measured on repositories with a median of 1,867 files — quoted as if it were typical. The real figure on ordinary projects is 22ms, and the table above now gives the whole distribution instead of one number.
• A table row built on a single prompt. One experiment, one prompt, generalised into a claim about when prepass does not help. Deleted, and replaced with what was actually measured across a corpus.

The second pass over the benchmarks was not a re-run of the same code. The harnesses were rewritten first — new failure accounting, new guards — and then the numbers were re-derived. Re-running identical code tests whether a machine is flaky. Running different code against the same definitions is what tests whether the claim was ever true.