Latent-Space Communication for Multi-Agent AI

In multi-agent systems today, models talk to each other through text, even though each one holds rich, high-dimensional internal states. On every hop, the sender compresses thousands of dimensions per token down to a sampled sequence of words and sends it. The receiver then runs a full forward pass to rebuild an internal state from scratch. Because of this, every hop costs twice: once in information (everything the sender represented but never put into words), and once in latency (token-by-token decoding is the slow part).

Recent research shows that the text step can be skipped. Interlat connects agents through their final hidden states and beats fine-tuned chain-of-thought exchange, even across different models ^[1]. Ramesh et al. go further and splice one model's activations into another's forward pass mid-computation ^[2]. Cache-to-Cache trains projectors that merge one LLM's KV-cache straight into another's, with better accuracy than text exchange at a fraction of the latency^[3]. All three are lab systems set up for specific models. A protocol would need to work for any pair.

This note works through what a general latent-space protocol would need to specify so that arbitrary models could talk this way. One distinction shapes every design choice that follows, and it comes down to access. With open-weight models, you can tap and inject at any layer. With a proprietary model behind a chat API, hidden states are out of reach, so the most a protocol can do is define the latent port a vendor might one day expose. The maths is the same in both cases.

Mismatched coordinate systems

Each model's hidden space has its own basis, set independently during training. Representations are only defined up to symmetries: neuron permutations, rotations, and scales that the normalisation layers absorb. With a paired corpus, aligning two spaces is roughly a Procrustes problem (closed-form if you stick to rotations, a learned map otherwise). Without pairings it becomes Gromov-Wasserstein optimal transport, which is much harder. There's also a separate estimation problem in working out which layer of model A corresponds to which layer of model B, because semantic depth doesn't scale linearly with layer index. The standard tool for choosing tap points is CKA over the layer-by-layer similarity grid ^[10].

Unusual activation geometry

Hidden states occupy only a small part of their ambient space. They cluster in narrow cones ^[11], and a handful of dimensions carry massive activations that act as internal bias machinery for the host model ^[12]. Naive cosine alignment breaks on both. So the adapter has to whiten against per-layer statistics, and it has to learn to leave the outlier dimensions alone, because those coordinates do real work inside the host model and carry no meaning for any other model.

Staying on the activation manifold

Mapping 4,096 dimensions to 8,192 is one linear layer, so the change in size is the easy part. The harder part is that valid activations live on a thin manifold inside the ambient space, and a translated vector that misses it produces undefined behaviour. At best that means garbage output, and at worst an accidental jailbreak. Because of this, the adapter needs manifold regularisation during training (a density model or discriminator penalising off-distribution output), and the receiver needs a runtime out-of-distribution gate. Section 05 reuses the same gate as a security control.

Where KV-cache exchange fits

A KV-cache is the most model-specific object in the whole stack. Keys and values are rotated by absolute position under rotary embeddings and shaped by head count, head dimension and grouped-query layout, so a cache is unreadable across architectures without per-layer, position-aware re-basing. Cache-to-Cache makes that work by training projectors for specific model pairs ^[3], and so far the published work covers only those specific pairs. So the protocol default is layer-output hidden states or pooled representations, and KV exchange stays a fast path inside fleets of identical models.

Continuous channels drift

Discrete tokens come with error correction built in: decoding snaps to the nearest vocabulary item, so small noise disappears at every hop. Continuous vectors have no lattice to snap to, so noise builds up along a multi-hop chain. Our proposed compromise is borrowed from VQ-VAE: quantise messages against a shared, versioned codebook ^[13]. This gives up a little fidelity in exchange for resynchronisation, compression and, as section 05 covers, a channel you can enumerate and audit.

The starting point for the threat model is that a transmitted hidden state is more sensitive than the text it encodes. Embedding inversion reconstructs text from embeddings with unsettling fidelity ^[14], and a full hidden state carries more than a pooled embedding. So latent traffic should be treated as plaintext-equivalent for confidentiality, at the very least. It also leaks model-internal structure, which turns an unmetered latent port into a free distillation oracle for whoever is listening. Encrypting the transport is the baseline, and the controls that matter most sit above it.

Inbound traffic is the more dangerous direction. Mechanically, a received latent is an activation-steering vector ^[15], and it passes every text-level safety filter because there is no text to filter. The defences reuse the correctness checks from section 03 as security controls:

• An out-of-distribution gate that rejects vectors outside the expected interlingua distribution.
• A norm clamp, so a single message can't overpower the host's own activations.
• Zero-initialised injection gates: injected latents start with zero influence and gain it only through training.
• Privilege separation: inbound latents are handled as data only. It's the same rule you'd apply to text, enforced at the injection site.

Outbound traffic is a leakage-control problem. The encoder is trained as an information bottleneck ^[16]: it sends the statistics sufficient for the declared task and destroys the rest, with an adversarial reconstruction attacker kept in the training loop as a check. Quantisation on the wire and rate limits on the port add to that.

The hardest problem is oversight. A latent channel removes the human-readable transcript, and a high-capacity continuous channel makes steganography easy, including agents colluding without their monitor noticing. The design offers four partial measures: shadow-decode every message into a logged text gloss, use the discrete codebook so the channel carries an enumerable vocabulary, run anomaly detection on the traffic, and require high-stakes actions to go through the auditable text channel. A capable enough sender can still evade all four, which is why oversight appears in the open problems in section 06.

The sections above are a design argument. The next step is to connect two real models and see where the design breaks, so that's what we're working on next: a small, simple testbed that turns the sections above into things you can run.

• Pick a pairing. Two open-weight models from different families, a Qwen and a Llama class model, small enough to sit side by side on a single workstation GPU.
• Hook the residual streams. Standard interpretability tooling already exposes activations at every layer. The testbed taps both models at matched depths, with section 03's CKA grid choosing the layers.
• Train the first bridge. A soft-token adapter on a paired activation corpus, with section 02's behavioural-distillation objective as the pass/fail signal.
• Make it one command. The whole rig packaged as docker-compose or a single CLI, so spinning up a local latent-space handoff between two real models takes minutes to start.

The whole thing will be open source, and we'll publish the evaluation criteria before any results. Four questions decide whether the idea holds up. Firstly, whether the latent handoff beats plain text on real tasks. Secondly, how much data each message costs. Thirdly, whether meaning survives a chain of hops. And finally, whether the security gates from section 05 stop a malicious vector. We'll link the repo here once it exists.