, 4 min read, voice-agents, livekit, architecture
A voice agent that knows when to stay quiet
Humans and an AI agent on the same LiveKit call: who gets to talk, how the agent takes notes, and why it is a participant, not a pipeline.
, 4 min read, voice-agents, livekit, architecture
Humans and an AI agent on the same LiveKit call: who gets to talk, how the agent takes notes, and why it is a participant, not a pipeline.
On a freelance project I built the voice side of a product where people and an AI agent share the same call. A person runs the conversation. The agent sits in the room, listens, takes notes, and speaks only when it is asked to or when it has something the person needs. Most of the interesting work was not the model. It was deciding who is allowed to talk, when, and what happens to everything that was said.
There are three moving parts, and each one does one job.
The browser never talks to the agent directly, and the agent never talks to the database directly. Both go through the room or through the server, which made every boundary easy to reason about and easy to test.
The first design most people reach for is a pipeline: audio in, transcript, model, audio out. That works for a one to one voice bot. It falls apart the moment there are two humans on the call, because a pipeline has no idea who is speaking or who they are speaking to.
Making the agent a real participant in the room fixed that. It subscribes to each person's track separately, so every line of the transcript already knows who said it. It can be muted, removed or added back like anyone else. And the humans see it in the participant list, which matters more than it sounds: people speak differently when they know the note taker is there.
A voice agent on a shared call has one hard rule: never talk over a person, and never answer a question that was meant for someone else.
So the agent has two modes. In listening mode, the default, it transcribes and takes notes and does not speak at all. It switches to active mode only on an explicit cue: someone addresses it by name, or the host presses a control in the UI that sends a message to the agent over the room's data channel. When it has answered, it drops back to listening.
The host can also take the floor back at any point. If a human starts speaking while the agent is talking, the agent stops mid sentence. Interruptions are the normal case in conversation, so they had to be cheap.
Taking notes is where the agent earns its place. It keeps a running transcript with speaker labels, and from that it maintains structured notes as the call goes on: decisions, open questions, action items with who owns them.
The notes are not written once at the end. The agent publishes updates to the room over the data channel while the call is live, so everyone can watch them form on screen and correct them while the context is still fresh. A wrong action item fixed during the call is worth far more than a perfect summary nobody reads afterwards.
When the room closes, LiveKit sends a webhook to the Fastify server. The server asks the agent's last notes to be finalised, stores them with the transcript, and makes them available to the rest of the product.
Create
the app asks the Fastify server for a room, and the server creates it and returns a token for each person
Dispatch
the server dispatches an agent into the room, so a worker joins as its own participant
Listen
the agent subscribes to every track and transcribes with speaker labels
Note
notes update live over the data channel as the conversation moves
Answer
when addressed, the agent switches to active mode, answers, and goes back to listening
Close
the room ends, the webhook fires, and the server stores the final notes and transcript
The agent is Python because that is where the audio and model tooling is most mature, and the LiveKit Agents SDK handles the parts that are easy to get wrong: joining rooms, track subscription, voice activity detection, interruptions and reconnects.
The server is Fastify because it is a small, fast HTTP layer with schema validation built in, and the rest of the product already lived in TypeScript. Keeping the control plane out of the agent meant agent workers could be scaled, restarted or replaced without touching anything a user calls directly.
Treat the agent as a participant with permissions, not as a feature bolted onto the audio. Make silence its default. Publish notes while the call is happening, not after. Each of those came from watching a real call go wrong, and each one made the agent feel less like a bot and more like a colleague who happens to write everything down.
Based in India. Works with teams in any time zone.
Comments