Research Evidence and Limits Behind the Advice to Have Coding Agents Fire Tracer Bullets First

If you have ever handed a coding agent an entire new feature to build, this scene will be familiar. For over 30 minutes without pause, the agent stacks up database schemas, API routes, and UI components layer by layer. Thousands of lines of code get generated, but when you actually click the button, nothing happens. Sometimes the screen shows nothing at all. Each individual layer looks plausible, but the moment you try to wire them together everything collapses, and finding where things went wrong takes longer than just starting the work over from scratch.

Two references that Modern Web Labs relies on prescribe the same remedy for this problem: Gene Kim and Steve Yegge's book Vibe Coding¹ and Matt Pocock's AI Hero course.²

(Image sources: Amazon, AI Hero)

The remedy is called the "tracer bullet." Yet neither reference cites empirical evidence explaining why this technique works especially well in the AI era. So Modern Web Labs verified it directly. We collected 21 source materials and validated 25 claims extracted from them. On top of that, we ran three rounds of cross-validation (deep research), and this report summarizes the results.

What is a tracer bullet?

The tracer bullet is a project management strategy introduced in The Pragmatic Programmer by Andy Hunt and David Thomas. A machine gunner firing in the dark does not aim by calculating ballistics. They fire luminous tracer rounds first, watch where the bullets actually go, then correct their aim.

(Tracer rounds fired during night live-fire training. Image source: Business Insider, photo: Sgt. Pete Thibodeau/US Marine Corps)

Applied to software development, the technique means first building a thin execution path that cuts through every layer of the system, confirming the architecture actually works, and only then filling in the flesh. In a 2003 interview, Pragmatic Programmer co-author Andy Hunt defined the core of the tracer bullet approach as "a skeleton application, with a thin thread of execution running end to end."³

(Image source: material presented by Modern Web Labs at the Megazone ICON 2026 event)

No experienced architect builds a house by fully decorating one room before moving to the next. They put up the frame first, verify that water, electricity, and plumbing run end to end, and only then finish the interiors. Tracer bullets follow that same sequence. In the software industry, the same concept also goes by names like walking skeleton, steel thread, and vertical slice.

Prototype development typically produces code that will be thrown away, because production code and prototype code differ in quality. Code built with the tracer bullet technique is different: it remains as the skeleton of the final system.

The bottom line: no paper has directly validated it

We could not find a controlled study that ran a coding-agent experiment under the banner of "tracer bullets." Broadening the search to synonyms like "walking skeleton" produced the same result. Stopping the investigation here would lead to the conclusion that the tracer bullet technique is "unfounded practitioner folklore." But that is not the case. Each of the components that make up the technique has quantitative evidence behind it.

Tracer bullets are a combination of three mechanisms.

• First, decide what to build and lay out the structure before writing any code (plan first),

• Second, stand up a working end-to-end skeleton first (skeleton-first decomposition),

• Third, look at the execution results and adjust the aim (iterative feedback).

Each of these three mechanisms has been quantitatively validated in peer-reviewed papers targeting LLMs (Large Language Models). Let's look at them one by one.

Evidence 1. Running, observing, and re-aiming raises success rates

The strongest body of evidence comes from research on iterative refinement. Instead of producing finished code in one shot, this approach generates an executable output first, then feeds the execution result back as feedback for another attempt.

Papers published at top machine learning venues point in a consistent direction. Reflexion (NeurIPS 2023) added a loop that has the agent verbally reflect on the cause of failure and apply that reflection to the next attempt, achieving pass@1 of 91% on HumanEval, a function-level code generation benchmark.⁴

(Image source: https://arxiv.org/abs/2303.11366)

Pass@1 refers to the rate at which generated code passes the tests on first submission, and 91% is 11 percentage points above the 80% that GPT-4, the state of the art at the time, was posting. Google DeepMind's Self-Debugging (ICLR 2024) had the model read the execution results of its own code and fix it, improving accuracy on the code translation (TransCoder) and basic Python (MBPP) benchmarks by up to 12 percentage points,⁵ and LeDex (NeurIPS 2024) trained the refinement process itself, lifting pass@1 by up to 15.92 percentage points.⁶ Self-Refine (NeurIPS 2023) showed similar gains, though it is worth noting that the benefit on coding tasks was 3.6 to 8.7 percentage points, smaller than the paper's overall average of about 20 percentage points.⁷

One caveat. These figures are all "up to" numbers, and the gains hold when there is external feedback such as test execution. That very condition is what ties this work back to tracer bullets, a point we will return to later.

Evidence 2. Planning first raises accuracy

The second line of research is plan-then-implement. Self-planning (ACM TOSEM 2024) showed across multiple benchmarks and languages that simply separating the planning phase from the implementation phase yields up to 25.4% Pass@1 improvement over implementation without planning, and up to 11.9% over Chain-of-Thought reasoning.⁸

Scale AI's PlanSearch (ICLR 2025) takes a different approach: instead of drawing multiple code samples directly, it first generates several branching solution strategies.⁹ The reason is that generating code first tends to produce answers that differ only in wording while sharing the same approach. So PlanSearch lists observations that can be drawn from the problem in natural language, combines them into distinct solution strategies, and then generates code for each strategy. With this method, Claude 3.5 Sonnet pushed the rate of getting at least one correct answer among 200 candidates (pass@200) on LiveCodeBench up to 77.0%. Repeated sampling of 200 code candidates without strategies scored 60.6%, and single-shot code (pass@1) scored 41.4%. The power of the plan-first approach stands out.

That said, PlanSearch's result assumes a large sampling budget, and the paper itself notes that for some models pass@1 can actually go down.

Evidence 3. The academic version of the skeleton-first approach: Parsel

The study that most closely mirrors the structure of tracer bullets is Stanford's Parsel (NeurIPS 2023). It is a "structure first, implementation later" pipeline: the task is first decomposed into a hierarchy of natural-language function descriptions to lock in the upper-level structure, and the implementation of each function is discovered through search over combinations that pass test verification.¹⁰

Results were measured on two benchmarks. On APPS, a benchmark of 10,000 algorithmic contest problems, decomposing the task first produced a relative pass rate more than 75% higher than generating code directly with AlphaCode or Codex (meaning that a baseline pass rate of 10% became at least 17.5%). On HumanEval, a function-level benchmark, combining the approach with automatically generated tests achieved pass@1 of 85%. The best previously published result was 67%.

APPS (Automated Programming Progress Standard) is a code generation benchmark released at NeurIPS 2021 by Dan Hendrycks and colleagues at UC Berkeley.¹¹ It consists of 10,000 problems collected from public algorithmic problem sites such as Codeforces. Like a contest that a human would solve directly, APPS requires reading a natural-language prompt and writing an entire program including I/O handling, and the generated code is graded by running test cases. Difficulty is divided into three levels: introductory, interview, and competition. Because it is much harder than HumanEval, where you complete a single function, it is well suited for confirming the effect of decomposition approaches on complex problems.

However, the 85% figure should not be taken at face value. The comparison point of 67% comes from prior work run under different conditions. The paper also ran a comparison that held the number of attempts equal while removing only the decomposition step. Under that setup, performance was 81.1%. In other words, the pure improvement attributable to task decomposition is about 4 percentage points. Adding the fact that the experimental models are from the 2022 to 2023 generation, Parsel is evidence pointing in the direction of a skeleton-first approach rather than a paper proving an overwhelming gap.

From a practitioner's perspective, this "attempts held equal" comparison actually provides an important implication. Raising the success rate by increasing the number of attempts stacks up token cost with every attempt. For organizations that bear token costs directly under an enterprise contract (metered pricing), task decomposition and planning, which raise the success rate for the same budget, are the rational choice. The fact that Parsel achieved a higher pass rate on APPS with a smaller sample budget points in the same direction.

The academic evidence up to this point is summarized in the table below.

Real-world evidence beyond benchmarks: Anthropic's "one feature per session"

Most academic benchmarks are function-level problems. The case closest to running agents against a large real codebase is Anthropic's report on long-running agent experiments.¹² In this experiment, the team used Claude Opus 4.5 and the Claude Agent SDK to build a claude.ai clone with more than 200 features across dozens of sessions. The report states that restricting the agent to "implement only one feature per session" was critical to preventing its tendency to build too much at once. Without the one-feature-per-session constraint, the agent would exhaust its context window mid-implementation, and the next session would start building on half-finished, undocumented features, repeating the failure. This is exactly the scene described in the introduction.

The official guides point in the same direction. Claude Code Best Practices recommends a four-stage workflow of explore, plan, implement, and commit, stating that "letting it code without a plan can produce code that solves the wrong problem," and explains that when you give the agent clear pass/fail signals such as test results or build exit codes, it can close the "work, check, repeat" loop on its own.¹³

(Image source: https://code.claude.com/docs/en/best-practices. The official document also covers the limitations of Plan mode, so a full read-through is worthwhile.)

OpenAI's Codex official documentation also recommends having the agent produce a plan before writing code for complex or ambiguous tasks, and has built a dedicated Plan mode (/plan) into the product.¹⁴ Note that neither company uses the term "tracer bullet," and these materials are qualitative reports based on internal experience rather than controlled experiments.

What the counterexamples actually tell us

There is research pointing the other way as well. Olausson et al. (ICLR 2024) showed that when compute budgets are held equal, the gains from self-repair can be marginal,¹⁵ and Huang et al. (ICLR 2024) showed that models have limits in correcting themselves purely through their own reasoning, without external feedback.¹⁶ Reading the shared conditions of these two counterexamples leads to an interesting conclusion. The gains disappear specifically "when there is no execution feedback."

That execution feedback is exactly what a tracer bullet creates from day one of the project. You need an end-to-end path for the code to run, running code is what produces external signals like tests, builds, and screens, and those signals are what make the benefits of iterative refinement possible. Code that is only stacked layer by layer cannot even execute before integration, so the agent has no feedback to receive.

The counterexample papers do not refute the tracer bullet approach. They explain why the skeleton has to be "executable" in the first place.

Remaining gaps

This insight report, which Modern Web Labs compiled from a range of papers, has its limits. Most of the academic evidence above rests on function-level benchmarks such as HumanEval, MBPP, and APPS, and on 2022 to 2023 era models. For agent work that spans an entire repository, no controlled experiment has yet compared the two approaches head to head. By the two approaches, we mean the tracer bullet method, which first builds and verifies a thin end-to-end skeleton and then adds features one at a time, versus handing the agent the entire feature set to implement in one shot. On repository-level benchmarks like SWE-bench, which asks agents to resolve real GitHub issues, this survey turned up no study that placed the two side by side.¹⁷ Whether the gains from an explicit planning step still hold in the latest reasoning models is also an open question.

The state of things as of Q3 2026 fits in a single sentence. No paper has proven the "term" tracer bullet, but each of its three constituent mechanisms has been quantitatively validated, and the direction aligns with field reports from the companies that have run agents the longest.

How to apply it

Modern Web Labs uses four practices in real coding-agent work, and we emphasize them in our training and consulting engagements.

• Slice tickets vertically. When handing a spec to an agent, don't split it into "all of backend, then all of frontend." Break it into thin units that cut through schema, API, UI, and tests in a single line. Matt Pocock designed his skill for decomposing specs into tickets so that each ticket takes the form of a tracer bullet.¹⁸

• Make "one feature per session" the rule. Size each feature so implementation can finish before the context window runs out. Anthropic's experiment showed that if context is exhausted mid-implementation, you are left with only half-built features. For a visual explanation of why an agent's attention degrades as tokens accumulate in the context window, see the Lab's Why Attention Scatters as Tokens Pile Up.

• Design pass/fail signals first. When you set up signals the agent can check on its own, such as tests, build exit codes, or screenshot comparisons, the gains demonstrated by iterative-refinement research actually kick in.

• Don't add flesh until the skeleton is verified. Confirm the tracer hits the target before firing live rounds.

In an era where coding agents perform much of the coding on behalf of humans, practitioner know-how accumulates faster than papers. The more that is true, the more finding an institution that knows which advice comes from which evidence becomes the criterion for choosing a good training provider and consultant. The tracer bullet technique is one piece of the practical know-how Modern Web Labs offers from that perspective.

References

1. Gene Kim and Steve Yegge, Vibe Coding: Building Production-Grade Software With GenAI, Chat, Agents, and Beyond, IT Revolution.

   ↩

2. Matt Pocock, Tracer Bullets: Keeping AI Slop Under Control, AI Hero.

   ↩

3. Andy Hunt and David Thomas, Tracer Bullets and Prototypes, Artima, 2003. A Bill Venners interview, the primary source for the tracer bullet concept.

   ↩

4. Reflexion: Language Agents with Verbal Reinforcement Learning, NeurIPS 2023.

   ↩

5. Google DeepMind, Teaching Large Language Models to Self-Debug, ICLR 2024.

   ↩

6. LeDex: Training LLMs to Better Self-Debug and Explain Code, NeurIPS 2024.

   ↩

7. Self-Refine: Iterative Refinement with Self-Feedback, NeurIPS 2023.

   ↩

8. Self-planning Code Generation with Large Language Models, ACM TOSEM, 2024.

   ↩

9. Scale AI, Planning In Natural Language Improves LLM Search For Code Generation, ICLR 2025.

   ↩

10. Stanford, Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions, NeurIPS 2023.

    ↩

11. UC Berkeley, Measuring Coding Challenge Competence With APPS, NeurIPS 2021.

    ↩

12. Anthropic, Effective harnesses for long-running agents, Anthropic Engineering, 2025-11-26.

    ↩

13. Anthropic, Claude Code Best Practices, Claude Code Docs.

    ↩

14. OpenAI, Codex Best Practices, OpenAI Developers.

    ↩

15. Is Self-Repair a Silver Bullet for Code Generation?, ICLR 2024.

    ↩

16. Large Language Models Cannot Self-Correct Reasoning Yet, ICLR 2024.

    ↩

17. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, ICLR 2024.

    ↩

18. Matt Pocock, 5 Agent Skills I Use Every Day, AI Hero.

    ↩