Voice Cloning Across Hosts: Making TTS Operational

I had a speech endpoint reporting healthy while the service behind it was unavailable. At first glance that makes no sense, but the endpoint was doing exactly what I had asked it to do. The bridge process was running, its HTTP listener was answering, and its configuration was readable. None of those checks told me whether the model had finished loading or whether its reference material was available.

That distinction mattered once speech requests started coming from an agent on another host. Most REST calls should finish quickly, while a local TTS engine may take considerably longer during a cold load. Raising every timeout would hide unrelated failures, but keeping the speech timeout short would make a working local model look broken. I eventually stopped treating text-to-speech as one opaque feature and began operating the request path as the small chain of services it really was.

Following that chain also forced me to be explicit about an ethical boundary. A voice alias is convenient configuration, not proof that anybody owns a sample, has permission to use it, or has consented to a generated voice. I limit this system to my own material, purpose-made synthetic references, or material I am explicitly authorized to use. What follows is an operations story, not a recipe for impersonation or a collection of voices to distribute.

That leaves a practical question: where does the reference come from? It does not come from this article. In my system it has to be a recording I made myself, purpose-made synthetic material, or a recording I have explicit permission to use. When I run a transcript-conditioned Qwen model such as Base or CustomVoice through my patched MLX-Audio Qwen handler, I also need a word-for-word transcript of that recording. I learned not to treat the transcript as a rough label for the clip. The recording and the words are one reference pair, and changing or omitting words can quickly spoil the result. Some supported models accept audio alone, so I check the input contract, format, useful sample length, and recording requirements for the model I am actually running.

Hermes Agent is the client at the other end of this particular chain. When I refer to the Dashboard or Gateway, I mean those Hermes components rather than generic parts of every agent framework. The names are specific, but the problem is not. Any agent that can call an OpenAI-compatible speech endpoint still needs clear ownership, useful health checks, tests from the real client process, and a recovery path I can explain when something fails.

The Agent-Facing Interface Needed to Stay Boring

The agent already knew how to call an OpenAI-compatible speech endpoint, and I wanted to keep it that way. It should not need to know which local engine happened to be running, where the model lived, or how that engine represented a cloned voice. Those details belonged behind the compatibility boundary.

The TTS Bridge accepts an OpenAI-compatible speech request, normalizes it, applies reviewed defaults, consults bounded discovery metadata, and forwards a compatible request to the engine. It can select an operator-defined alias, apply a small pronunciation map to the target text, translate model-specific controls, and normalize an unsupported OGG or Opus request to WAV. When it substitutes a format, it reports the requested and delivered formats in response headers instead of quietly pretending it returned something else.

The resulting ownership split is fairly simple. The engine owns model loading, inference, generated media, and the reference registry; the bridge owns the stable agent-facing contract. They now run together on the inference host under the same service identity, and the bridge reaches the engine over loopback. The agent host is just a client, with no model runtime, sample filesystem, or fallback bridge to maintain.

Open full-size diagram

Boundary	What it owns	What it does not prove
Agent client	Input text, requested alias, and expected response format	Which reference was used or whether generated speech is acceptable
TTS Bridge	Alias discovery, request normalization, strict control validation, compatibility translation, redacted diagnostics, and bounded forwarding	Model readiness, voice ownership, consent, or audio quality
TTS engine	Model loading, inference, generated audio, capability reporting, and the reference registry	That the operator reviewed the semantic match between a recording and transcript
LLM-Ops-Kit	Desired state, typed dependency, lifecycle plan, health observation, and reviewed logs	Generic component relocation or autonomous recovery

The generalized provider contract and third-party provider recipes remain deferred. A tested OpenAI-compatible request boundary does not make every local or cloud speech service a supported provider.

The Hands-On companion turns this boundary into a model-free executable test bench using a teaching bridge, fake capability and registry endpoints, generated tone audio, and invented text.

The Pair Belongs Where It Is Consumed

My first bridge configuration mapped a neutral alias to an audio path and a matching transcript path. That worked, but it exposed an awkward cross-host truth: a filesystem path has meaning only to the process that reads it. Sending a path from one machine to another does not make the file appear there, and checking it on the client tells me nothing about the engine’s namespace.

I learned that lesson during recovery from an unrelated host failure. External storage holding model and reference material was unavailable, yet the bridge could still run and the system could still return audio. What it had not established was that the intended clone-reference path was ready. Direct and bridged clone-reference checks passed after storage was restored. I cannot honestly claim that the host failure caused every poor result seen during that period, or that a transcript mismatch occurred. I can say that process health and returned audio were not strong enough acceptance tests.

The correction was architectural rather than another path check. The inference service now owns each accepted audio file and transcript as one immutable registry record. Registration gives the pair an opaque reference_id and records both hashes atomically. The bridge maps an operator-friendly alias such as narrator to that identifier, leaving neither the agent nor the bridge with a reason to know the raw paths or transcript content.

This is a much better cross-host contract:

Contract	Preferred use	Remaining limit
Registered reference ID	Normal alias-driven cloning across the service boundary	Hashes establish identity, not that the transcript semantically matches the recording
Inline reference object	Explicit bounded request using authorized material	Sensitive content crosses the request boundary and requires stricter handling
Legacy server paths	Allowlisted compatibility with an existing deployment	Paths remain host-specific and are disabled unless an operator explicitly permits roots

The semantic match still needs a human ear. A digest can prove that I am using the same audio and text I reviewed earlier, but it cannot listen to a recording and certify that the transcript says what was spoken.

Discovery Became Part of Readiness

Co-location removed the most fragile filesystem boundary, but I still did not want the bridge operating blind. It refreshes engine capabilities and registry metadata, then reuses that result for up to 30 seconds. The cache retains capability data plus registry reachability and count, not the registry’s reference IDs. Alias-to-ID mappings still come from the bridge’s configured aliases, while the engine remains authoritative for whether an ID is valid.

The cached capability family and supported-control list let the bridge reject an incompatible request before synthesis. When a refresh cannot reach either discovery endpoint, it replaces the cached result with an unreachable state and the request fails closed. A request may reuse still-current metadata between refreshes, so this is a bounded cache rather than a fresh discovery round trip on every synthesis call. Reference-ID validity is left to the engine instead of being guessed from a bridge-side copy of the registry.

Diagnostics follow the same boundary. Target text, reference text, sample paths, and inline audio are redacted. An opaque reference ID may remain for correlation, but it must not be published. Operational records can still say that discovery succeeded, which compatibility branch was selected, and whether the upstream request completed. They do not need to become a second archive of private speech material.

Pronunciation translation is deliberately one-sided. A configured replacement such as reading a punctuation character by name applies to the new text being synthesized. It must never rewrite the accepted reference transcript. The reference pair is immutable because changing either half changes the cloning input.

Hands-On 9A makes both failure paths executable in Fail Before Synthesis.

Health Still Has More Than One Layer

The wrapper reports the bridge process, listener, bridge HTTP health, configured upstream, capability revision, registry reachability, and request result. Those signals answer different questions.

Signal	Question answered	What it cannot establish
Bridge process	Is the compatibility process running?	That the engine is reachable
Bridge HTTP health	Can the bridge answer and expose bounded metadata?	That a registered alias can synthesize audio
Capability and registry discovery	Does the bridge understand the current engine contract and see references?	That the model is loaded or the output sounds right
Speech request	Can the complete path return valid audio within its bound?	Speaker similarity, intelligibility, or consent
Human listening review	Is the output acceptable for the intended use?	Future stability or recovery behavior

I resisted the easy fix of making every timeout large. The bridge uses a bounded upstream timeout for synthesis, while a client can give speech a longer allowance without relaxing ordinary API and liveness requests. When something fails, I can check the bridge, discovery, engine, and synthesis layers in order instead of repeatedly increasing one global number and hoping for the best.

Retries need the same discipline. Retrying a connection failure after the engine becomes reachable is different from restarting an engine after a crash. Replaying synthesis can also duplicate work after the caller has gone away. Current recovery remains manual and reviewed, not an autonomous restart loop.

The lab also separates bridge health from request success after the fake upstream stops.

The most misleading failure in this deployment happened outside that chain. Direct requests from an operator shell reached the relocated bridge, but requests from the Hermes Dashboard did not appear in the bridge log at all. macOS unified logs showed NECP denying Local Network access to the Dashboard’s LaunchAgent process. NECP is the macOS network-policy subsystem involved in enforcing Local Network privacy decisions. The resulting No route to host looked like an IP-routing failure even though the bridge, route, and API were working.

That incident changed the client lifecycle rather than the bridge topology. LLM-Ops-Kit now manages the Dashboard as a standalone background process, while the Gateway remains a launchd component. The bridge remains co-located with the engine and reaches it over loopback. Correct network permission and a real Dashboard-originated synthesis request were required before I called the repair accepted. A generic GET /v1/models was not useful evidence because the bridge does not implement that route; its relevant observations are /health, /v1/audio/voices, and POST /v1/audio/speech.

This is the broader rule: test network access from the exact process, user, and security context that will make the production request. A successful terminal command proves the endpoint is reachable from that terminal. It does not prove that a LaunchAgent, sandboxed application, container, service account, or background worker has the same permission.

Hands-On 9A turns that lesson into a production-mapping checklist without pretending the model-free fixture tested macOS networking or process supervision.

The Runtime Is Part of the Product

The clean cold-start tests exposed a familiar macOS service problem: a command that worked in an interactive terminal failed when started noninteractively. Bare python did not reliably select the product environment, and audio-format work depended on a media executable that was available in the terminal but absent from the managed process path.

The durable repair was to make the runtime explicit and immutable. In this deployment, the accepted engine reports patched MLX-Audio 0.5.0+unixwzrd.1 from a UV-managed product environment. That version is a dated fact about this deployment, not a recommendation for everybody else. The important part is that the exact runtime passed clone-reference, long-text, streaming-recovery, malformed-request recovery, and model-isolation checks before promotion. A package version and a generic synthesis smoke test would not have told me any of that.

LLM-Ops-Kit’s Python remains a separate ownership boundary from the engine runtime. The bridge and engine may be co-located without sharing an accidental shell environment. That separation makes upgrades and rollback more explainable than one large development environment activated through a profile.

Automatic Recovery Is Still a Proposal

When an engine is unstable, automatic restart is tempting. A naive KeepAlive policy would make things worse by turning an intentional stop into a fight with the service manager, restarting rapidly during an outage, and hiding the difference between one crash and a persistent failure.

The recovery design under consideration is opt-in and has not been deployed. It requires desired-running state, bounded backoff, a visible restart count, the last exit status, the last successful recovery, and a retry budget. An explicit operator stop changes desired state and suppresses recovery. A crash while desired state remains running can enter the bounded recovery path. Exhaustion remains visible instead of being disguised as a process that restarts forever.

Open full-size diagram

This figure is design intent, not current-state documentation. Adapter-owned recovery policy, outage gating, counters, and recovery evidence remain deferred and unaccepted.

Telemetry Controls Belong in Managed Configuration

The TTS environment includes libraries that can integrate with Hugging Face services and other reporting systems. The reviewed controls belong in managed component configuration, not in an interactive shell:

HF_HUB_DISABLE_TELEMETRY=1
DO_NOT_TRACK=1
WANDB_DISABLED=true

Those settings reduce common reporting paths. They do not prove that a process has no outbound traffic. That claim needs source and dependency inspection plus network observation or a deny-by-default egress test.

I also avoid using HF_HUB_OFFLINE=1 as a telemetry shortcut. Offline mode changes model and artifact loading. It should be enabled only after every required artifact is cached and offline loading has passed acceptance.

Current State

The deployed service now co-locates the TTS Bridge and patched MLX-Audio engine on the inference host. LLM-Ops-Kit manages the bridge as a standalone background component through its dedicated adapter rather than through launchd; its explicit restart_policy=never preserves intentional stops. The engine owns immutable registered audio and transcript pairs and remains authoritative for reference-ID validity. The bridge maps configured neutral aliases to opaque IDs, caches capabilities plus registry reachability and count for a bounded interval, validates controls strictly, performs model-specific compatibility translation, and redacts target text and reference content or paths from diagnostics. The agent host is a client only, and its Dashboard is now a separately managed background process after the LaunchAgent network-context failure.

The focused bridge source suite passes all eighteen tests. A real agent request through the deployed bridge returned valid audio after direct, alias, inline-reference, registered-reference, long-text, streaming-recovery, malformed-request recovery, and isolated-model canaries passed. Those results establish protocol and operational acceptance for this deployment. They do not establish universal provider compatibility, semantic transcript verification, consent, or subjective voice quality.

Legacy path pairs remain available only as explicitly allowlisted compatibility behavior. Generic component relocation and automatic recovery are not implemented. The cutover that moved these services was a reviewed manual operation, not a host-field edit or a component move command. The subsequent stack restart also required explicit Dashboard and tunnel starts after those components did not return automatically, so a successful stack command is not yet sufficient acceptance evidence for this topology.

Next Work

I am less interested in adding more voices right now than in preserving this contract when things fail. Final-artifact repetition still has to cover cold start, outage, restart, rollback, and client reconnection. Human listening review remains separate from objective WAV and protocol checks, and the public provider contract still needs a clear local and remote boundary that does not ship voices, credentials, or reference material.

The proposed recovery policy also needs implementation and failure injection before it can move out of a diagram. It must preserve explicit stops, use bounded cadence, expose every attempt, distinguish discovery failure from a process crash, and stop when its retry budget is exhausted.

The separate Hands-On companion models the preferred registered-reference contract without using a real voice. Its fake upstream exposes capability and registry discovery, its neutral alias forwards only an opaque ID, and its failure cases prove that a failed metadata refresh and an unsupported control stop before synthesis. One synthetic path pair remains as labeled legacy compatibility coverage. That is enough to make the boundary executable without pretending a generated tone measures voice-cloning quality.