When I began looking for ways to make my agents less expensive and more efficient, I expected a shortlist of products. Instead, I had memory systems, caches, gateways, context compressors, shell-output reducers, and observability platforms open in different tabs, each presenting a number that sounded like the answer.

They were not answering the same question. A memory engine that retrieves an old decision is not competing directly with a provider cache that reuses computation for an unchanged prefix. A shell-output reducer is not doing the same job as a proxy that changes model context. An observability tool may save no tokens and still be the best first addition because it shows where the waste is.

Once I stopped treating the landscape as one product category, the evaluation became practical. I was no longer asking which tool had the largest advertised percentage. I was asking which layer was causing the problem, what evidence the tool would change, how I would know it helped the whole task, and how cleanly I could remove it if it did not.

First, Separate the Layers

My first landscape document included a ranked table. It was useful as a starting hypothesis in July, but its scores were estimates, not benchmark results, and I do not think it would be honest to republish that table as a current leaderboard. The more durable part of the work was the taxonomy beneath it.

Layer What it tries to improve Characteristic failure
Durable knowledge Keeps reviewed material in a human-readable authority The document exists but is poorly classified, stale, or improperly shared
Agent memory and semantic retrieval Selects useful prior material for the current task Irrelevant, stale, duplicated, or unauthorized context enters the task
Provider prompt caching Reuses computation for stable repeated prompt prefixes Dynamic prefixes prevent cache hits or accounting is misunderstood
Response caching Replays a previous answer without inference A superficially similar request receives a stale answer
Context compression Changes the prompt so less reaches the model Important evidence or instructions disappear
Shell-output reduction Shortens tool results before the agent reads them A failing assertion, traceback, ordering detail, or diff hunk is hidden
Routing and gateways Selects endpoints and may add budgets, fallback, or caching The gateway becomes another critical failure domain or changes model behavior
Observability Measures tokens, latency, cost, errors, and behavior Private prompts or responses are retained or exported merely to obtain metrics

This distinction changed the order in which I was willing to experiment. Native prompt caching can reduce repeated-input cost without changing the text the model receives, while observe-only measurement can reveal waste without rewriting prompts or answers. Those are easier changes to justify than lossy compression, semantic answer reuse, global memory hooks, or a new proxy in every model path.

Memory and caching also stopped looking like substitutes. I want Main Vault and LLM-Wiki to remain the durable, human-readable authority for reviewed knowledge. Mnemosyne can build useful facts, vectors, and retrieval structures from eligible material, but those are derived state that I must be able to rebuild. A provider cache may then reuse computation for the exact prompt prefix that still needs to be sent. Each layer can help without pretending to own the others.

The Landscape Became an Evolution Story

The dated snapshots matter because the environment changed while I was studying it. In the July comparison, the local Mnemosyne installation looked partial or stale from the shell being inspected, while its Hermes integration was the reason it remained the strongest first memory candidate for that agent. MemPalace was more interesting as a bounded Cursor or Codex experiment because of its cross-tool transcript workflow, but it retained source text verbatim and introduced a heavier storage and embedding footprint. The recommendation was deliberately conservative: do not run both during one canary, and do not enable either globally simply because persistent memory sounds useful.

Later project evidence changed the Mnemosyne picture. Integrity, doctor, rebuild, and general/private routing checks were recorded after the early comparison, and the August 16 inspection found mnemosyne-memory 3.15.1 with mnemosyne-hermes 0.5.0 in the active Hermes environment. The gateway was stopped during that inspection, so the observation proves the installed integration rather than a memory request flowing at that moment. That is still a meaningful progression: uncertain installation, then bounded acceptance evidence, then an installed but idle snapshot. It is not the same as declaring every retrieval path continuously validated.

RTK followed a similar path through several versions. The lifecycle document captured the operating rule that mattered most to me: use reduction selectively and retain a raw bypass for failing tests, complete diffs, exact snapshots, and ordering-sensitive logs. By August 16, RTK 0.45.0 was installed and directly reported telemetry disabled. That tells me the version and one current control; it does not tell me the realized savings for every agent, whether every hook is active, or whether filtering improved task success. A compression percentage is incomplete evidence if the agent has to rerun the command to recover what was removed.

Headroom is the clearest example of why configuration, installation, acceptance, and current routing must remain separate claims. It had passed an isolated health and OpenAI-compatible request check in the August 14 product history. On August 16, version 0.35.0 was installed, but its desired and observed lifecycle state was stopped, its launchd job was not loaded, and its local listener was absent. Hermes still named the local endpoint in configuration, but a configured URL does not carry traffic when the service and gateway are stopped. The accurate current statement is less exciting and much more useful: installed and previously validated, available as a possible local path, but stopped and not carrying model traffic at inspection time.

The diagram below is the picture I now use instead of a leaderboard. Solid paths represent a layer with dated deployment or acceptance evidence. Dashed paths represent candidates, optional routing, or work that is not currently carrying traffic.

A status-explicit tool landscape showing reviewed Vault authority projecting to tested Mnemosyne memory or alternative memory candidates, an explicit choice between selective RTK and raw output, provider-managed prefix reuse, and Headroom installed but stopped and bypassed.
Open full-size diagram

Correctness and Disruption Are Gates

I use a weighted score, but I do not let the average excuse a dangerous failure. Correctness preservation carries the largest weight, followed by disruption risk and realized savings. Operability, compatibility, privacy and local control, observability, and project health fill out the comparison. The arithmetic is useful after a candidate survives the basic questions: Did the task still succeed? Did the agent retain the evidence needed to explain a failure? Did the change introduce a duplicate action, a stale answer, or an unexplained behavioral difference?

A tool that hides a failing assertion is rejected even if it halves the visible output. A response cache that replays an answer after repository state changes is rejected even if it avoids an entire inference. A memory system that lowers history tokens but adds stale facts, extraction calls, retries, and latency has not necessarily optimized anything. These are not minor deductions in a score; they are reasons to stop the canary.

Here, a project canary is a small trial with easy rollback; a soak is a longer observation period under realistic use. One candidate is compared with a baseline, evidence is collected, and correctness and disruption are checked before the canary proceeds to soak. The rejection path stays visible because bypass and uninstall are part of the design, not an admission that the experiment failed.

An optimization adoption gate moving one candidate from whole-task evidence through correctness and disruption checks, a project canary, and a risk-sized soak, with explicit stop, bypass, rollback, or uninstall paths after rejection, canary failure, or soak regression.
Open full-size diagram

The project therefore measures whole-task economics rather than one counter. The minimum record includes the task and repository revision, candidate version, cold or warm state, outcome, regression evidence, input and output tokens where available, cached input, injected memory, tool-output volume, cost where it can be measured honestly, tool calls, duplicates, retries, elapsed time, and resource use. Memory evaluations add returned, relevant, stale, and contradictory items. If subscription tooling does not expose reliable token or dollar accounting, I record the observable behavior rather than inventing precision.

Question Weak evidence Evidence I can act on
Did it save tokens? A vendor percentage or one command’s estimate Baseline and candidate totals across the same representative tasks
Did memory help? A plausible retrieved note Better task outcomes with measured precision, stale rate, and injected context
Did it preserve correctness? The final answer looked reasonable Regression evidence, exact failure details, and behavioral differences retained
Is it operationally cheap? Installation succeeded Upgrade, bypass, rollback, backup, diagnostics, and removal all exercised
Is it private? A telemetry-off flag or local-storage claim Versioned source, dependency, configuration, retention, and network-behavior audit

The initial decision record set practical thresholds around those ideas. Observe-only tools can begin with a short canary. Output reducers need a raw bypass and a longer comparison. Memory, routing, response caching, and prompt compression stay project-scoped until representative tasks and rollback have been exercised. A replacement should offer a material score or cost improvement without increasing disruption. Those are decision gates, not claims that a current candidate achieved them.

Complementary Layers Beat a Universal Product

The resulting stack is intentionally modest. Main Vault and LLM-Wiki preserve durable reviewed knowledge. Mnemosyne is a tested derived-memory layer for the Hermes environment, with a coarse bank boundary that remains separate from the proposed fine-grained policy model discussed in the previous article. RTK can reduce predictable shell noise while the raw command remains immediately available. Provider-native caching is worth measuring when stable prefixes actually recur. None of those components should claim to be the complete optimization system.

The deferred candidates still have useful roles. MemPalace may be worth a project-only cross-tool memory canary, but it was not installed at the dated review and its verbatim retention creates a larger privacy obligation. Mem0 remains a general-purpose alternative rather than a reason to replace a working integration. Langfuse could provide observe-only tracing, while LiteLLM could eventually add local accounting, budgets, or routing. Both require retention, redaction, backup, health, bypass, and removal plans before they belong in the path. Semantic response caching and prompt compression remain higher-risk because a repository task can look similar while its correct answer has changed.

This is where uninstall cost became one of my favorite criteria. Installation demos are plentiful; clean removal evidence is not. If a tool modifies hooks, owns a database, inserts a proxy, downloads models, or captures transcripts, I want to disable it, handle its data deliberately, remove only what it owns, and prove the agent still works. A tool that cannot leave cleanly has a higher disruption cost before it fails.

Privacy Is Part of Performance

Observability can become its own data-leak path. Prompt and response traces are excellent debugging material precisely because they may contain private instructions, repository details, retrieved memory, tool output, and model-generated artifacts. The project keeps raw metrics private and publishes only redacted aggregates. Local counters are not automatically outbound telemetry, but placing an observer or gateway in the request path changes what must be inspected and retained.

The July telemetry audit established a method I still trust: record exact versions, inspect source and dependency manifests, inspect managed environments and opt-out controls, observe outbound connections with models already available, deny unrelated egress during acceptance where practical, and distinguish required cloud-provider traffic from product telemetry. What I cannot do is stretch that dated audit across later upgrades. RTK’s disabled telemetry setting was directly checked on August 16; the broader stack still needs a post-upgrade source, configuration, dependency, and network-behavior audit. “Telemetry-free” would be a much larger claim than the evidence supports.

Research Needs a Cadence, Not Constant Churn

New releases and new products appear faster than I can evaluate them properly, which creates a temptation to swap components whenever a headline number moves. I would rather review the landscape weekly or biweekly, collect candidate changes in one place, and run a canary only when the expected improvement is large enough to justify the work. That keeps research moving without turning the operating environment into a permanent migration exercise.

The recurring report is still an operational gap. The project documented the cadence and scheduled reporting direction, but the named evidence does not contain a completed recurring landscape report. I am not going to describe a process as established merely because a job was configured. The useful next milestone is a real dated report built from a common workload, followed by another report that shows whether the recommendation remained stable.

Current State

At the August 16 inspection, RTK 0.45.0 was installed and directly reported telemetry disabled. Selective reduction with a raw-output bypass remains the operating rule, but this article does not claim a current universal savings figure or complete hook coverage. Headroom 0.35.0 was installed and had earlier isolated acceptance evidence, but it was stopped, unloaded, not listening, and not carrying model traffic. Mnemosyne 3.15.1 and its Hermes integration 0.5.0 were installed; earlier project records contain integrity, doctor, rebuild, and coarse general/private routing acceptance, while the stopped gateway meant the inspection did not exercise a live memory request.

Main Vault and LLM-Wiki remain the durable reviewed knowledge authority, while Mnemosyne remains rebuildable derived state. MemPalace, Mem0, Langfuse, LiteLLM, semantic response caches, and prompt compressors remain candidates or deferred layers rather than globally adopted replacements. The older estimated ranking is retained as a dated starting hypothesis, not presented as a current scorecard.

Next Work

The next useful artifact is a common-workload report rather than another catalog. It should compare the baseline with one candidate at a time, retain exact regression evidence, measure whole-task behavior, and publish only redacted aggregates. The stack also needs a post-upgrade telemetry and network-behavior audit, and every upstream version and capability mentioned here must be rechecked before publication.

I also need to make the research cadence real. One completed weekly or biweekly report, followed by a second comparable report, would provide better evidence than constantly revising a product ranking. If that work produces a complete, reproducible harness rather than a collection of private scripts and traces, it may become the Hands-On companion to this installment. Until then, the most important optimization remains the least glamorous one: change one layer, preserve the baseline, and keep the way back obvious.