2026-11-05

Master of Puppets: Voice-Driven Agent Development

I know how this sounds. If someone told me about this setup over coffee, I’d probably nod politely, take a step back and wonder when they last slept eight hours. I’ve spent months putting together tunnels, multi-agent multiplexers, headless runtimes and canvas apps. And I’ve made peace with it: this workflow is probably not for you. It’s definitely not for anyone who wants a sane setup, best practices or a calm afternoon.

It’s an aggressive experiment. I remove everything that sits between a thought in my head and a command in a terminal. No keyboard. No clicking through IDE tabs. Just voice, headless terminals and lots of agents that do the actual work.

The problem with voice-driven development has always been the plumbing. You can’t just yell at your laptop and expect it to write code, especially when you’re nowhere near your desk.

My day starts with an iPad running Allume¹. It’s an infinite canvas app with good handwriting support, sticky notes and its own CLI/MCP hooks. Next to it I have Claude Code in a voice conversation on my phone or tablet. But voice sessions can’t reach the local tools or sockets on my machine at home.

I didn’t want to open my workstation to attacks. So I set up a Cloudflare Tunnel that goes straight to my local MCP server. Opening your machine to the outside world is scary, so the whole handshake sits behind OAuth that Anthropic supports. On top of that there’s a strict passphrase, which I pull from my local keychain. The voice agent can’t do whatever it wants on the system. It can only talk to a dispatcher and start specific sub-sessions.

The dispatcher sits behind the MCP layer. It checks incoming requests, keeps the security boundaries and coordinates actions. When something has to actually happen, the dispatcher talks to RACE. RACE is my terminal multiplexer, and it has a big MCP interface. There’s a link to it in the footer.

With RACE an agent can start any number of terminals, write text and send raw key events. It can also read the scrollback buffers and check exit codes. So voice gives the orders, the dispatcher checks the boundaries and RACE does the work.

Now, imagine you let an LLM start terminals whenever it wants. With standard Git and one branch, running that in parallel turns into a disaster in fifteen seconds. Three agents that edit the same files will wreck your staging area.

That’s why I dropped standard Git for this workflow and use Jujutsu (jj) instead.

Jujutsu has workspaces, and they’re what make lots of agents working at the same time possible. Every time a new task starts, the dispatcher opens a terminal and also creates a separate Jujutsu workspace for that task.

Say I’m dogfooding Comedit, an editor which I’ve been building in OCaml. I run it, I try the open command on an empty buffer and the file doesn’t load. I don’t alt-tab, I don’t open an editor and I don’t set breakpoints. I’m walking across the room, and I just say into the microphone:

“Found a bug in Comedit. Open a new terminal. Notice that running open followed by a file name in an empty editor fails to open the file. Spin up a Claude session in the Comedit group, track down the root cause, and fix it.”

The voice layer picks up the sentence and sends it over the tunnel to the dispatcher. Then the machine goes to work:

1. RACE opens a new terminal tab.
2. Jujutsu creates a new, separate workspace (jj workspace add).
3. Claude Code starts in that terminal and gets the context. It looks at the OCaml code that parses the AST and starts tracing the bug on its own.

No lock contention. No dirty working copies. Ten or twelve terminals can run different experiments on the codebase at the same time. None of them knows about the others until it’s time to merge.

The hardest part of voice isn’t the audio. It’s latency and asymmetry. When you talk to a model, you expect an answer right away. You can’t sit on an open voice line in silence for six minutes while an agent builds OCaml across a matrix and runs regression tests.

So the backend doesn’t make the voice channel wait. Every time I send a command or check in over voice, the server returns a JSON payload with a BTW (by the way) field.

The BTW field is a status feed that comes along with whatever I asked. If I say “Hey, check the status of the workspace,” I get the direct answer. On top of that, the dispatcher adds the background jobs which changed state recently:

{
  "status": "ok",
  "data": { "active_workspaces": 14 },
  "btw": [
    "Comedit terminal #14: isolated parser bug, wrote failing test case, patch applied.",
    "Comedit terminal #11: test suite passed without regressions."
  ]
}

The voice model reads the BTW field and puts it into the spoken answer: “Status is clear, you’ve got fourteen workspaces up. By the way, the bug you flagged in workspace 14 has a patch ready, and the test suite in workspace 11 just finished clean.”

Once you have 150 or 200 terminals in different workspaces, you can’t remember where anything is. You can’t grep hundreds of sessions in your head.

That’s where the other tool integrations come in:

• Canvas Coordination: While I talk, the voice assistant can put new sticky notes, cards and diagrams on my Allume canvas on the iPad. I sketch ideas or cross off tasks with the stylus while the agents work.
• Auto-Focusing: I don’t look through terminal groups to find where a task ended up. I just tell the voice model: “Focus the terminal working on the Comedit open bug.” RACE moves the infinite canvas to that terminal window and brings it to the front.
• Review Handoffs: When an agent finishes an implementation and runs a code review, it picks out the URL of the pull request or diff. Then it opens the browser and takes me right to the changes. I never have to search for a link.

It sounds slick, but it has problems.

First, the system can be stubborn. The dispatcher has its own prompt boundaries. Sometimes it pushes back or refuses a request outright, because it gets a boundary wrong. For example, it once decided on its own that it shouldn’t start a terminal outside a group whitelist. I had to stop and debug the logic of the agent.

Second, you have to watch code quality very closely. When you give agents whole implementation loops, they’ll happily write code that works but drifts away from your architecture. My job changed almost completely. I don’t type syntax anymore. I review diffs, look for edge cases and keep the agents on track.

And third, the part nobody talks about, is the exhaustion.

I used only this workflow for three days straight. You can grab your phone, put the leash on the dog, walk through the park and do PR reviews, new sessions and patches only by voice. You dictate a review: “Look at that PR, check if the edge cases around empty buffers are covered, draft a comment asking for more test coverage, and stage it.” It feels amazing.

Then you get home, sit down and realize your brain is completely fried.

Software engineering doesn’t get easier when you remove typing. Typing gives you natural breaks. Your thoughts slow down while your fingers catch up. Without typing, you spend four hours straight on 20 parallel threads of architecture decisions. You take notes on an iPad, sort through incoming BTW updates and review code at full speed. It wears you out. It isn’t lighter work. It’s a lot more work, done in a fraction of the time, and it costs your brain a lot.

I’ve talked to colleagues at work about this, and they pretty much all agree: they look at me like I’ve lost my mind. They call me a hopeless workaholic. They tell me they have zero need for a swarm of terminals that they control by voice.

And honestly, maybe they’re right. But it’s also frustrating, because I know how cool this is. You watch a whole codebase change, repair and deploy itself when you speak, and it feels like magic.

The loneliest part is that I can’t easily share it. Claude Code’s voice interface only works in English right now. If I want to show my wife what I’ve been building all these late nights, I have to speak English to a computer in our living room. That makes the whole thing feel distant and artificial. If she could hear me talk to it in Polish and watch the screens react, maybe it would make sense to her. Instead it stays my own little world. It lives between me, my headset and a wall of terminal scrollback.

Keyboards and mice had a great run. But I watch terminal sessions organize themselves on an infinite canvas when I speak, and the old tools start to feel like they’re from another era. I think we’re heading toward a future without them. Even if, right now, it’s an exhausting and slightly lonely place to build in.