Fast Agents Are Just Agents That Don't Wander

Here's a result from our benchmark that we didn't expect.

We asked an AI agent to fix a rendering bug in bubbletea, a Go terminal UI library. We gave it expert-written context about the project — the kind of brief a strong engineer would write for a new teammate. We ran it four times.

The results: 21 seconds. 31 seconds. 44 seconds. 252 seconds.

Same agent. Same model. Same context. Same bug. Three runs found the fix fast. One run wandered for over four minutes.

Then we ran the same task with sourcebook-generated context — a structural map of the codebase extracted automatically from import graphs and git history. Four runs: 34 seconds. 34 seconds. 41 seconds. 57 seconds.

Not faster on the best run. But the bad run never happened.

This pattern repeated across our benchmark. And it changed how we think about what sourcebook actually does.

The experiment

19 real, closed GitHub issues across 10 open-source repos in 4 languages. Each task has a merged PR fix. The harness checks out the codebase to the state before the fix, validates the bug is still present, injects a context file (or none), and runs the agent.

Model: Claude Sonnet 4 (pinned). Max turns: 50. No human intervention. Then repeat runs on the top 5 tasks — 4 runs per condition — to measure variance.

Three conditions:

• None — just the issue description and the codebase.
• Handwritten — a CLAUDE.md written by a developer who understands the project.
• sourcebook — auto-generated structural context. Hub files from import graph analysis, co-change coupling from git forensics, convention signals. No stack labels, no project structure, no standard commands — only what agents can't discover by reading files.

Repos: cal.com, hono, pydantic, fastapi, gin, chi, bubbletea, clap, next.js, vercel/ai — spanning enterprise monorepos, Python libraries, Go web frameworks, and Rust CLI tools.

What the single runs showed

In the initial benchmark (one run per condition), sourcebook was faster than handwritten context on 13 of 19 tasks. The average gap was 16%. On first glance, this looks like a speed story.

*clap #6275 no-context: agent produced 0 lines (gave up). **fastapi #14454 handwritten: agent edited the wrong file (0% file overlap with reference PR).

But single runs lie. We knew that. So we ran the top 5 tasks four times each.

What the repeat runs showed

The speed advantage mostly disappeared. No individual result was statistically significant at 95% confidence. The variance within each condition is too high for N=4 to detect a reliable speed difference.

But something else appeared in the data.

Look at the range column. sourcebook produced the tighter range on 4 of 5 tasks. The means are close. The variance is not.

That bubbletea result — 21 to 252 seconds with handwritten context — is a 12x spread. Same agent, same brief, same bug. The agent sometimes found the right path immediately. Sometimes it wandered for four minutes.

With sourcebook, the range was 34 to 57 seconds. The bad run never happened. Not because the agent got lucky. Because the structural map — hub files from import graph analysis, module boundaries, where the rendering pipeline lives — gave it a path that doesn't depend on luck.

The mechanism

Agents don't fail because they're slow. They fail because they don't know where to start.

Point an agent at a 10,000-file monorepo and tell it to fix a bug. It will read files. Lots of files. It will trace imports, scan directory trees, try to build a mental model from raw source. Sometimes it finds the right file on the second try. Sometimes it reads 30 files before it gets oriented. That's the variance. That's where the 252-second runs come from.

Handwritten context helps — on average. But human-written briefs describe how a framework works in general. They say things like "tests are in __tests__" and "we use tRPC for API routes." That's useful. But it doesn't tell the agent which file is the structural center of the rendering pipeline, or which files always change together, or where the import graph converges.

Structural context is different. It's a map. It says: this file is imported by 52 other files. This file and that file change together 88% of the time. The rendering logic converges here. An agent with a map doesn't explore. It navigates.

That's the mechanism. Not speed. Removal of wasted motion.

Three things the data proves

By language

Structural context helps most where the module system is implicit and agents need a map to navigate.

The ETH Zurich question

In February 2025, ETH Zurich found that auto-generated context hurts agent performance by 2–3%. That finding was right — for the kind of context they tested: stack descriptions, project structure, standard commands. Things agents can figure out by reading files.

Our data shows the flip side. Auto-generated context that surfaces non-discoverable structural signals — import topology, co-change coupling, hub files — doesn't hurt. It performs at parity with handwritten context. And it eliminates the failure modes that handwritten context can't prevent.

The problem was never auto-generation. It was what you auto-generate. Redundant information creates noise. Structural information creates a map.

Where it doesn't help

Small, well-documented TypeScript libraries. When the issue description alone is sufficient and the codebase has strong type declarations, agents navigate fine without a structural map. hono (+69%) and vercel/ai #13354 (+65%) both showed sourcebook adding overhead without value.

Simple, localized bugs. If the error message points directly to the fix, context is overhead. The value of a map scales with how lost the agent would be without one.

The pattern: sourcebook's value scales with structural opacity. Large monorepos, implicit module systems, unfamiliar languages — that's where agents need a map. Small, readable projects don't need one.

Limitations

• No individual result reached statistical significance. Agent behavior has high intrinsic variance. N=4 runs per condition is enough to see the variance pattern but not enough for p<0.05 on any single task. We're reporting what the data shows, not what we can formally prove.
• Correctness is file overlap, not semantic comparison. We compare which files the agent touched against the reference PR. This doesn't verify the logic of the fix.
• Single model (Claude Sonnet 4). Results may differ on other models.
• We wrote the handwritten baselines ourselves. A different expert might write better ones. The comparison is against a reasonable handwritten brief, not the theoretical best.

What we learned

We started this benchmark trying to prove sourcebook makes agents faster. The data told us something different. sourcebook makes agents more reliable. It eliminates the wandering runs. It prevents the wrong-file failures. It matches human-written context without human effort.

Over time, reliability is speed. The team that runs 100 tasks where every one finishes in 30–57 seconds ships faster than the team that's sometimes 21 seconds and sometimes 252. Not because the agent got faster. Because it stopped getting lost.

A map, not a boost.