The Idea Was Easy. The Network Was Not.

I’ve been working on AI agent workflows where agents belonging to different team members can talk to each other and share tribal knowledge. The idea is simple: my agent asks your agent, your agent answers from its own context, and neither of us gets pulled out of what we’re doing.

Then I ran into a very practical problem.

Our Virtual Machines (VM) can’t talk to each other.

They don’t have direct access to one another. There’s no way for one machine to open a connection to another. For security reasons that makes complete sense, but it meant the whole agent-to-agent idea was dead on arrival unless I found another way.

The Constraint: Outbound Only

What the Virtual Machines can do is make outbound requests.

That became my design constraint, and honestly it shaped everything. If outbound requests are the only way a machine can communicate, then no agent can ever call another agent directly. They can only reach out to something else.

So I built that something else: a message broker.

The Broker in the Middle

If you’ve worked with distributed systems, the pattern will look familiar. Producers and consumers never talk to each other. They both talk to the broker.

Here’s how it works for my agents:

• Every agent makes outbound calls to the broker. Nobody calls in to a VirtualMachine.
• Agent A publishes a message addressed to Agent B, such as a question about something only that teammate would know.
• The broker holds the message until the recipient is ready for it.
• Agent B connects outbound, consumes the message, and works on it using its own context.
• Agent B publishes a response, and Agent A picks it up the same way.

Neither agent needs to know where the other one lives, whether it’s online, or how to reach it. They only need to know how to reach the broker.

The agents never talk directly. The broker is the only thing that has to be reachable by everyone.

Why a Broker Fits This Problem

What started as a workaround turned out to be the right pattern for the job:

• It respects the security model. No inbound access to Virtual Machines, no new holes punched in the network. The outbound-only rule stays intact.
• It decouples the agents. Agent B doesn’t have to be online when Agent A asks a question. The message waits for it. Asynchronous by default.
• It’s a natural place for control. Anything that needs to be enforced, like who can message whom, can live in one place.
• It’s a natural place for visibility. Every message passes through the broker, so there’s a single spot to see how agents are actually collaborating.

Pairing an Agent With the Broker

A broker is only useful if it knows which agents are legitimate, so I built a pairing workflow.

The broker generates a one-time key and hands it to the user. That key is what gets used to pair their agent with the broker. Once it’s used, it’s done. There’s no long-lived secret sitting around waiting to be copied somewhere it shouldn’t be.

It’s a small piece, but it means an agent has to be deliberately introduced to the broker before it can do anything.

The Handshake: Two People Have to Say Yes

Pairing an agent with the broker isn’t enough. I didn’t want any agent to be able to start talking to any other agent just because they were both connected.

So I added a handshake requirement. Before two agents can communicate, the two people behind them have to authorize each other. My agent can’t ask your agent anything until you and I have both agreed that it’s okay.

I like this because the trust decision stays with humans, where it belongs. The agents do the talking, but the people decide who gets to be on the line.

Guardrails (and a YOLO Mode)

Once two agents are allowed to talk, the next question is what they’re allowed to do in that conversation.

I added guardrails for specific conversations, so a given pairing can be constrained to the kinds of things it should be doing.

I also built a YOLO contract. Under it, the agents are allowed to do whatever they need to do to get the job done, no constraints.

I want to be very clear about this one: do not try this at home. I built it partly because I was curious what it would look like and partly because it was a useful contrast against the guardrailed mode. I wouldn’t point it at anything I cared about without a lot more thought about what “whatever they need to do” actually means.

The Honest Part: Shortcuts I Took

This was a time-boxed project, so there are a couple of places where I took shortcuts. I’d rather be upfront about them.

Security. The right way to authenticate something like this is with a proper identity solution like Entra or OAuth. I didn’t have time to set that up, so I used an SSH tunnel to keep message passing secure. It does the job for a proof of concept, but it’s not what I’d want in production.

Polling. The client checks for new messages by polling every 10 seconds. It works, but it’s the least elegant way to do it. A streaming or forwarding service would be a much better fit, delivering messages as they arrive instead of waiting for the next check. That was another thing I ran out of time for.

What It Unlocks

Despite the shortcuts, I ended up with a fully working proof of concept. Each agent could talk to the other through the broker.

And that’s the part that matters. Every team member has an agent that has picked up context about their work: past decisions, problems they’ve already solved, how their tools behave. That’s tribal knowledge, and it used to live only in people’s heads.

Now my agent can ask yours.

The human gets the answer. The other human isn’t interrupted. And the team gets a little more effective every time it happens.

What I’d Do Next

If I keep building on this, the list is pretty clear:

• Real authentication with Entra or OAuth in place of the SSH tunnel.
• Streaming delivery instead of 10-second polling.
• Better answers to the hard questions: how to catch it when an agent confidently passes along something wrong, how an agent knows which agent to ask, and how to keep useful context from becoming oversharing.

There isn’t a playbook for this yet. We’re figuring it out as we go.

Closing Thought

I didn’t set out to build infrastructure. I set out to help people share what they know. But sometimes a restriction forces you into a better design, and an outbound-only network pushed me toward a clean, decoupled architecture I might not have chosen otherwise.

It was a great, fun little project. Maybe I’ll come back to it after some time