Latent-Space Communication for Multi-Agent AI

Here's the thing about multi-agent systems today: two models that each hold rich, high-dimensional internal states talk to each other through... text. Every hop does the same wasteful dance. The sender compresses thousands of dimensions per token down to a sampled word sequence, ships it, and the receiver burns a full forward pass rebuilding an internal state from scratch. You pay twice: once in information (everything the sender represented but never put into words), and once in latency (token-by-token decoding is the slow part).

The interesting bit: research keeps showing the dance is optional. Interlat wires agents together through their final hidden states and beats fine-tuned chain-of-thought exchange, even across different models ^[1]. Ramesh et al. go further and splice one model's activations into another's forward pass mid-computation ^[2]. Cache-to-Cache trains projectors that merge one LLM's KV-cache straight into another's, with better accuracy than text exchange at a fraction of the latency^[3]. All three are lab systems wired up for specific models. None of them is a protocol.

That's the gap this write-up digs into: what would a general latent-space protocol need to specify so that arbitrary models could talk this way? One distinction drives every design choice that follows. With open-weight models, you can tap and inject at any layer. With a proprietary model, you can't touch hidden states through a chat API at all, so the best a protocol can do is define the latent port a vendor might one day expose. Same maths in both worlds. Very different access.

Nobody agreed on a coordinate system

Two models' hidden spaces don't share a basis, and nothing in training makes them. Representations are only defined up to symmetries: neuron permutations, rotations, and scales that the normalisation layers absorb. With a paired corpus, aligning two spaces is roughly a Procrustes problem (closed-form if you stick to rotations, a learned map otherwise). Without pairings it degrades to Gromov-Wasserstein optimal transport, which is a much worse day. And picking which layer of model A corresponds to which layer of model B is its own estimation problem, because semantic depth doesn't scale linearly with layer index. CKA over the layer-by-layer similarity grid is the standard tool for choosing tap points^[10].

The spaces themselves are weird

Hidden states don't fill their ambient space. They huddle in narrow cones ^[11], and a handful of dimensions carry massive activations that act as internal bias machinery for the host model ^[12]. Naive cosine alignment breaks on both. The adapter has to whiten against per-layer statistics, and it has to learn to leave the outlier machinery alone: those coordinates are load-bearing at home and gibberish abroad.

Off the manifold, off the map

Mapping 4,096 dimensions to 8,192 is one linear layer; that part is easy. The catch is that valid activations live on a thin manifold inside the ambient space, and a translated vector that misses it produces undefined behaviour. Best case, garbage output. Worst case, an accidental jailbreak. So the adapter needs manifold regularisation during training (a density model or discriminator punishing off-distribution output), and the receiver needs a runtime out-of-distribution gate. Hold that thought: section 05 turns the same gate into a security control.

Why not just ship the KV-cache?

Because a KV-cache is the most model-specific object in the whole stack. Keys and values are rotated by absolute position under rotary embeddings and shaped by head count, head dimension and grouped-query layout, so a cache is unreadable across architectures without per-layer, position-aware re-basing. Cache-to-Cache makes that work by training projectors for specific model pairs^[3]; nothing published generalises it. So the protocol default is layer-output hidden states or pooled representations, and KV exchange stays a fast path inside fleets of identical models.

Continuous channels drift

Discrete tokens come with free error correction: decoding snaps to the nearest vocabulary item, so small noise vanishes at every hop. Continuous vectors have no lattice to snap to, so noise compounds along a multi-hop chain. The compromise we like is borrowed from VQ-VAE: quantise messages against a shared, versioned codebook^[13]. You give up a little fidelity and get back resynchronisation, compression, and (this matters in section 05) a channel you can enumerate and audit.

Start with the premise that changes everything: a transmitted hidden state is more sensitive than the text it encodes, not less. Embedding inversion reconstructs text from embeddings with unsettling fidelity ^[14], and a full hidden state carries more than a pooled embedding. So treat latent traffic as plaintext-equivalent for confidentiality, and then some: it leaks model-internal structure too, which turns an unmetered latent port into a free distillation oracle for whoever is listening. Encrypting the transport is table stakes. The interesting controls live above it.

Inbound is the scary direction. A received latent is, mechanically, an activation-steering vector ^[15], and it sails past every text-level safety filter because there is no text to filter. The defences are section 03's correctness machinery, promoted to security controls:

• An out-of-distribution gate that rejects vectors outside the expected interlingua distribution.
• A norm clamp, so no single message can out-shout the host's own activations.
• Zero-initialised injection gates: influence is earned during training, never assumed.
• Privilege separation: inbound latents are data, never instructions. The same rule you'd want for text, enforced at the injection site.

Outbound is a leakage-control problem. Train the encoder as an information bottleneck ^[16]: send statistics sufficient for the declared task, destroy the rest, and keep an adversarial reconstruction attacker in the training loop to keep it honest. Add quantisation on the wire and rate limits on the port. And call it what it is: attack-resistance engineering. Nothing here is a provable privacy guarantee, and if this page claimed one, its own build would fail.

The deepest problem is oversight, and we won't soften it. A latent channel deletes the human-readable transcript, and a high-capacity continuous channel is a gift to steganography, including agents colluding right past their monitor. What the design can offer is partial: shadow-decode every message into a logged text gloss, use the discrete codebook so the channel is an enumerable vocabulary rather than a firehose of floats, run anomaly detection on the traffic, and hard-require that high-stakes actions surface through the auditable text channel. A capable enough sender can still evade all of it. That sentence belongs in the threat model, not in the small print.

A design argument only gets you so far. At some point you have to wire two real models together and watch what breaks. So that's what we're working on next: a small, deliberately unglamorous testbed that turns the sections above into things you can run.

• Pick a pairing. Two open-weight models from different families, a Qwen and a Llama class model, small enough to sit side by side on a single workstation GPU.
• Hook the residual streams. Standard interpretability tooling already exposes activations at every layer. The testbed taps both models at matched depths, with section 03's CKA grid choosing the layers.
• Train the first bridge. A soft-token adapter on a paired activation corpus, with section 02's behavioural-distillation objective as the pass/fail signal.
• Make it one command. The whole rig packaged as docker-compose or a single CLI, so spinning up a local latent-space handoff between two real models costs a coffee break, not a weekend.

The whole thing will be open source, and we'll publish the scorecard before the scores. Four questions decide whether the idea survives contact with reality: does the latent handoff beat plain text on real tasks, how much data does each message cost, does meaning survive a chain of hops, and do the security gates from section 05 actually stop a malicious vector? Once the repo exists, it gets linked right here.