The first number looked encouraging enough that I nearly took it at face value. In one synthetic comparison, Headroom reduced the input from 1,324 tokens to 1,181, a 10.8 percent reduction, and the model still produced the exact answer I expected. On paper, that looked like exactly what I had been trying to accomplish.

Then I looked at the rest of the result. The response had slowed from 5.77 seconds to 9.14 seconds, and the completion had grown from 244 tokens to 258. This was one test on July 14 using Headroom 0.31.0, not a benchmark suite and certainly not a promise about a later version. What it gave me was a much better question: had I improved the agent, or had I merely improved one number?

What I actually cared about was whether the agent completed the task correctly, retained the evidence needed to explain a failure, and remained understandable when something went wrong. I also needed to know whether the extra component earned the latency and operational work it introduced. Token reduction still mattered, but only inside that larger result.

A Smaller Prompt Is Not Automatically a Better Agent

I had been lumping several very different mechanisms together under the word optimization. Once I separated them, the problem became easier to reason about. Provider-managed prefix reuse can avoid recomputing an exact block of input without changing the text the model receives. RTK can reduce predictable shell noise before it occupies the agent’s context. Headroom can change broader model-visible context in the request path. Memory may replace a long history with a smaller retrieved subset, while a response cache may avoid inference altogether.

Those savings are not interchangeable. Their failure modes are different, and so is the evidence I need before I trust them. A cache can replay an answer that was correct before the repository changed. A shell reducer can hide the one assertion that explains why a test failed. A context optimizer can save tokens while adding latency or disturbing a provider’s own prompt-cache locality. Memory can reduce history and then inject a stale fact that sends the agent down the wrong path.

That is why I stopped asking, “How many tokens did it save?” as the first question. I now start with, “Did the task still work, and can I prove why?” A candidate that fails that test does not get rescued by an impressive reduction percentage.

Fix the Starting State Before Comparing Anything

“Run the same task twice” sounds simple, but the word same gets slippery very quickly. By the second run, files may have changed, the model or provider cache may be warm, a memory system may have learned from the first pass, and the agent may take a different route for perfectly ordinary reasons. If I do not record those conditions, I can easily congratulate the optimizer for a difference it did not cause.

For each comparison, I need a sanitized workload definition and a fixed or recorded starting state. That includes the repository revision or fixture identity, the candidate and model versions, the available tool surface, the route, whether caches and models are cold or warm, the memory state, and the run order. Where I cannot hold a condition fixed, I record it and alternate or repeat the order rather than pretending it disappeared.

The baseline and candidate must also retain comparable evidence. If the raw path keeps a full traceback while the optimized path keeps only a summary, I cannot compare correctness after the fact. I can compare two outputs only when both preserve enough detail to apply the same pass, fail, and diagnostic criteria.

I want the control path to be boring. Boring makes it possible to explain what changed:

A paired optimization canary in which one sanitized workload and fixed starting state feed a raw baseline and exactly one candidate path. Both retain comparable correctness evidence and content-free metrics before a decision gate, scoped canary, and risk-sized soak, with stop, bypass, rollback, or removal paths after any failure.
Open full-size diagram

This is not meant to imply that one pair of runs proves anything statistically. It is the minimum honest shape of the experiment. The same workload enters the raw baseline and one candidate path; both produce correctness evidence and content-free measurements; only then does the candidate reach a scoped canary and a longer soak under realistic use. A mismatch, canary failure, or soak regression goes to the same place: stop, bypass, roll back, or remove the candidate.

RTK Worked Best Where the Noise Was Predictable

The RTK example was almost comically lopsided. One failing test collection produced 8,936 bytes of raw output and a 102-byte reduced view, with an estimated token saving of 90.6 percent. That figure belongs to one noisy failure shape, not to Python testing in general. More importantly, the full raw failure had been retained separately, so the reduced view did not become the only surviving evidence.

The same evaluation did practically nothing useful for already concise Git and ripgrep output. I was glad to have that result because it broke the idea that every command should pass through the same machinery. A reducer can only remove redundancy that is actually present, and it still adds another behavior I have to understand when the output looks odd.

My working rule became selective rather than global. Predictable directory listings, repeated progress output, and other well-understood noise may be reasonable candidates. Assertions, tracebacks, snapshots, complete diffs, ordering-sensitive logs, and unfamiliar failures stay raw. During regression analysis I would rather spend more tokens than lose the line that explains the defect. The bypass is part of the feature, not an emergency escape hatch added after something goes wrong.

Headroom Produced Two Honest and Very Different Numbers

This is where the attractive 10.8 percent number really lost its authority. It was not the only Headroom observation that day. Across a separate set of eight requests, the recorded aggregate was 96 input tokens saved out of 80,193, or 0.12 percent. Both results can be true because the workloads were different, and neither sample is large enough to describe typical performance.

July 14 observation Workload boundary What it showed
1,324 to 1,181 input tokens; 10.8% One synthetic comparison on Headroom 0.31.0; exact answer preserved; latency rose from 5.77s to 9.14s The request contained reducible context, but savings alone did not clear the whole-task gate
96 of 80,193 input tokens; 0.12% Aggregate over eight separate observed requests Realized savings can be negligible when the request shape offers little safe reduction

The table is not a performance comparison between two versions or two products. It is evidence that the same layer can look very different as the workload changes. It also explains why I do not carry these percentages forward to Headroom 0.35.0. The later version passed isolated health and OpenAI-compatible POST routing acceptance on August 14, but compatibility acceptance did not reproduce the July performance experiment.

By the August 16 inspection, Headroom 0.35.0 was still installed but both desired and observed stopped. Its launchd job was unloaded, its listener was absent, and the route bypassed it. It was not carrying model traffic. That current state does not erase the earlier test; it keeps installation, compatibility, measured performance, and active routing as four separate claims instead of one vague statement that the tool “worked.”

Provider Caching Can Change the Meaning of a Win

There was another wrinkle in the July evidence. One final no-tool turn reported 26,320 cached tokens out of 26,711 input tokens, rounded there to a 99 percent prompt-cache observation. That is one dated turn, not a general cache rate, but it shows why cache state belongs in every comparison. A large exact prefix may be cheap and fast when the provider can reuse it, while a smaller rewritten prompt may lose that reuse and cost more despite looking better in an input-token column.

This interaction is easy to miss when each layer reports its own preferred counter. A context optimizer sees the tokens it removed. A provider reports cached input. A memory system reports retrieved items or injected context. A shell reducer estimates output reduction. None of those counters, by itself, tells me what happened to the complete task.

I therefore record cold and warm state, cached input where the provider exposes it, prompt and completion tokens, and the route actually used. I avoid assigning causality when the accounting is incomplete. If a subscription tool or local endpoint does not expose reliable token or dollar figures, the honest result is “not observed,” not a fabricated estimate with decimal places.

The Measurement Record Has to Follow the Whole Task

I do not need to save a private conversation just to prove that an experiment ran, and I do not want a cost dashboard to become the only record of what happened. The useful middle ground is a content-free record that still answers the engineering questions.

Question Content-free evidence to retain
Did the task succeed? Workload or fixture ID, starting revision, pass/fail result, regression identifiers, and behavioral differences
What context changed? Input and output tokens, cached input, injected-memory tokens, tool-output volume, and candidate version
What did it cost operationally? Elapsed time, model and tool calls, retries, errors, resource use, and cost where the accounting is reliable
Was the comparison controlled? Raw or candidate route, run order, cold/warm state, cache state, model version, and tool-surface identity
Can I get back? Bypass result, rollback result, retained baseline, data-handling decision, and removal verification

Hashes can identify fixtures and outputs without publishing their contents, although a hash does not prove two outputs are semantically equivalent. Correctness still needs workload-specific checks: test results, expected fields, invariant checks, or reviewed behavioral differences. For debugging tasks, the retained raw failure is often more valuable than the final answer.

The same record also exposes indirect costs. An optimizer may reduce model input but add retries. A memory layer may reduce history and make an extraction call. A gateway may lower provider cost but add latency, a new database, and another process that can fail. Whole-task measurement prevents those expenses from falling outside the frame.

Local Metrics and Outbound Telemetry Are Different Questions

Telemetry is another place where the language gets sloppy. I want usage counters, latency records, error classes, and route information, but I do not want prompts, responses, repository details, or retrieved memories sent to another service simply to obtain them. Local operational metrics are not outbound telemetry merely because they measure usage. They become a data-egress issue when a configured integration transmits them elsewhere.

A telemetry-off setting is useful evidence, but it is only one layer of evidence. The stronger audit records the exact software and dependency versions, inspects source and dependency manifests, checks managed configuration and opt-out controls, reviews retention behavior, observes outbound connections with required artifacts already available, and tests whether required operational endpoints still work when unrelated egress is denied. Required cloud-provider traffic also has to be separated from analytics rather than counted as either harmless or suspicious by default.

RTK 0.45.0 directly reported telemetry disabled during the August 16 check. That supports the narrow statement about what the installed command reported at that time. It does not prove current hook coverage, realized savings, or a telemetry-free environment. The broader stack changed after the July 19 audit, so a complete post-upgrade source, dependency, configuration, retention, and network-behavior audit remains outstanding.

Every Optimizer Has to Pay Rent

I have come to think of every optimization layer as having to pay rent. Even when it behaves correctly, a local binary adds installation, update, and hook behavior. A proxy adds process ownership, dependencies, health checks, routing, logs, timeouts, and a bypass. A cache adds invalidation and retained data. A memory layer adds retrieval quality, deletion, provenance, and stale-state questions. Those obligations are part of the price of the saved tokens.

The decision gate is therefore asymmetric. Correctness failure ends the experiment immediately. A passing comparison earns only a project-scoped canary with an explicit bypass and rollback. A passing canary earns a risk-sized soak. Retention comes after the candidate demonstrates a material whole-task benefit without unacceptable disruption, not after it produces one attractive counter.

This approach is slower than installing every promising tool globally, but it makes reversibility normal. It also gives negative results a useful place in the record. “No useful reduction on concise output” and “saved tokens but added latency” are not failed experiments. They are the information that prevents a narrow optimization from becoming a permanent source of ambiguity.

Current State

At the August 16 inspection, Headroom 0.35.0 was installed but stopped, unloaded, not listening, bypassed, and not carrying model traffic. Its August 14 health and live POST checks remain dated compatibility acceptance, while the only publishable performance observations in this article belong to Headroom 0.31.0 on July 14. RTK 0.45.0 was installed and directly reported telemetry disabled; no global hook was enabled, and the historical reduction result remains limited to one noisy failing-test collection with raw evidence retained.

The project has a measurement policy and several useful point observations, but it does not yet have a completed common-workload benchmark report that justifies a universal savings figure or a replacement decision. Selective reduction, direct routing, raw-output bypass, and exact regression evidence remain the conservative baseline.

Next Work

The next useful artifact is a sanitized paired-run harness with fixed fixtures and content-free output. It should compare direct and optimized routes, alternate run order, record cold and warm state, preserve exact regression evidence privately, and produce a redacted record of tokens, provider-cache fields, injected memory, tool-output volume, latency, errors, retries, resource use, and cost where it is genuinely observable. The harness should exercise comparison refusal, canary failure, soak regression, bypass, rollback, and removal rather than treating the happy path as sufficient.

The complete stack also needs a post-upgrade telemetry and network-behavior audit covering source, dependencies, managed configuration, retention, and observed connections. A Hands-On companion can follow when the harness and sanitized fixtures actually exist and pass private acceptance. Until then, the most defensible token optimization is not the one with the largest isolated percentage. It is the one that improves the complete task, preserves the evidence, and leaves the agent easier to trust instead of harder.