How I designed a system for AI agents to handle phone calls

Talking to an AI agent over the phone seems like a natural extension of a chatbot: someone asks a question, the system answers, and the conversation continues. But there is much more behind that experience than turning a model into a voice. There is telephony, real-time audio, transcription, turn-taking, security, session state, tools, and a way to reconstruct what happened after the call ended.

In this article, I want to explain how I approached the design of a system in which an agent handles incoming calls. This is not a recipe tied to one company or credential. The providers mentioned are the ones used in this implementation, but each layer can change. What matters is understanding the responsibilities and decisions that keep the solution maintainable.

The central idea is to separate two things that may initially look like one: handling the call and observing it. The first must have the lowest possible latency. The second must make it possible to know what the system transcribed, what the agent answered, which tools it executed, and which data was confirmed, without making real-time audio fragile.

The path of an incoming call: the carrier starts the session through a webhook, the Gateway manages the audio, and the agent processes each turn before returning the response through the same stream.

The problem is not only turning text into speech

A call does not arrive as text. It comes from the telephone network and must become a conversation that the agent can understand and answer naturally. A minimal architecture needs to answer at least these questions:

• How does the call enter through the telephone carrier?
• How does audio reach the server and how does the response come back?
• When is the person considered to have finished speaking?
• When should the agent be queried?
• What happens if the person interrupts synthetic speech?
• How is risky input filtered before it reaches the model?
• How can the system leave evidence of what happened without storing promises it never fulfilled?

In my implementation, the call enters through a telephony provider. That provider invokes an HTTPS webhook and opens a bidirectional audio stream over WebSocket. A Gateway receives the audio and manages the technical state: call admission, turns, playback, interruptions, and shutdown.

The specific telephony provider can change. The tunnel or mechanism that exposes the webhook can change as well. The relevant decision is to preserve the contract: a public endpoint to start the call and a bidirectional audio channel that the voice engine can manage.

The flow working today: STT, agent, and TTS

The first version uses a modular pipeline. The person's audio passes through STT (speech-to-text), the agent reasons over the transcript and produces a text response, and a TTS (text-to-speech) service turns it back into audio.

In this case, recognition and synthesis are handled by separate voice services. ElevenLabs Scribe is currently used for STT and ElevenLabs Flash for TTS, while Twilio acts as the carrier. They could be replaced by other providers as long as the audio, transcription, and synthesis interfaces remain stable.

The simplified flow is:

1. The person calls the number.
2. The carrier sends the start webhook and opens the Media Stream.
3. The Gateway receives telephone audio over WebSocket.
4. The STT engine transcribes the stream and detects the end of the turn through VAD (voice activity detection).
5. When a transcript.final exists, the guardrails review the input.
6. The agent receives the allowed text, its context, and the tools it is meant to use.
7. The model response passes through the output guardrails.
8. TTS synthesizes the approved text and the Gateway sends it back through the audio stream.

The call does not pass through a browser. Nor does an additional layer need to touch the audio to record telemetry. The Gateway's voice plugin owns the transport, the technical session, barge-in, and shutdown.

The modular pipeline separates STT, guardrails, the agent, and TTS. It also shows the realtime alternative, which reduces steps but changes turn-taking, interruption handling, and validation.

Why start with a modular pipeline?

Because it makes the control points visible. The final transcript is a concrete piece that can be reviewed before calling the agent. The text output can also be validated before it is spoken. STT, the model, and TTS can be evaluated or replaced independently.

That does not mean it is the only alternative. A real-time voice architecture can work more directly from audio to audio, which is often called speech-to-speech or STS. In that mode, a realtime provider integrates perception, conversation, and voice generation in one session.

This can reduce the number of hops and make the interaction feel more fluid. But it is not a cosmetic change: it modifies the turn model, interruptions, latency, access to the transcript, and where guardrails are placed. That is why I treated it as a configurable alternative rather than an automatic replacement for the modular pipeline. Before adopting it, it should be compared against the same call script, measuring quality, safety, barge-in, and time to first audio.

Guardrails: one layer before listening and another before speaking

In a call, unsafe input does not become safe because it is spoken. The transcript may contain attempts to manipulate the agent, out-of-scope requests, or information that should not reach a tool. On the other side, a response may leak data, promise an action that does not exist, or be inappropriate to say to someone over the phone.

That is why the flow has guardrails at two points:

• Input: after the final transcript and before the agent.
• Output: after the model response and before TTS.

This placement is deliberate. The input guardrail sees what the system understood from the person, and the output guardrail reviews exactly the text that will be converted into speech. If a validation fails, the policy must define a safe response or block the turn. It is not enough to assume that the agent's prompt will solve the problem.

This separation makes it possible to review the input before it reaches the agent and the response before it becomes speech. A phone conversation may feel natural, but it still needs controls against malicious instructions, out-of-scope requests, and responses that expose data or promise actions that were never executed. Voice adds immediacy, but it does not eliminate the risks of an agentic system.

A voice agent should not inherit every internal tool

Another important decision was to separate the reception agent from the internal agent. The first has its own workspace, limited context, and explicit tools. The second can retain capabilities used for internal operations, but those capabilities should not be exposed by default to someone who calls.

This is a practical application of least privilege: the call should be able to solve its purpose without exposing tools or knowledge it does not need. If an external action is needed in the future, that action deserves its own tool, validations, and separate audit trail.

The system may detect, for example, that the person wants to speak to a human, that the case is out of scope, or that essential data is still missing after several clarification attempts. The detection tool records that fact idempotently, but it does not transfer the call, create a ticket, promise a callback, or notify anyone. Detecting a need and executing a consequence are two different responsibilities.

That distinction prevents a common assistant failure: the model saying “I already escalated it” when no external integration actually ran.

A call's state is more than a conversation

A voice conversation can contain useful data, but free text is not enough to operate reliably. That is why I added a dedicated tool to record structured fields during the call.

For example, the agent may detect a name, phone number, reason, or time slot. When the person mentions it, the value is recorded as detected. Only after the person explicitly confirms that exact value does it become confirmed.

missing → detected → confirmed
             │
             └── detected (if the person corrects the value)

The detail matters: silence, failure to correct, an agent summary, or the end of a flow do not count as confirmation. If someone corrects their phone number, the history is preserved instead of overwriting the previous value. This makes it possible to know what was heard, what was confirmed, and in which turn it happened.

The tool does not insert data into a CRM or trigger a commercial action. Its current responsibility is to validate and leave structured evidence. When an integration is connected, it should be another explicit and verifiable action.

There is also a plugin for the skill lifecycle. The agent activates a skill when it matches the call's intent and records a terminal result when it completes, is cancelled, or fails. The evidence is not something the model said: it is the actual execution of the tool. That distinction makes a trace useful for auditing instead of just being a story about what the model believes it did.

Escalating to a person without breaking the conversation

There are situations in which the best response from an agent is to make room for a person: when the caller asks for it, when the case is out of scope, when essential information is still missing after clarification, or when a business rule marks a particular priority.

The escalation plugin turns that situation into a structured event with a reason, priority, and summary. From there, an orchestration layer consumes the event and executes a single escalation action per call. Idempotency matters: if the agent detects the same case again or the process is retried, it should not create multiple transfers, tickets, or notifications for one conversation.

The destination does not need to be unique. Depending on the policy configured for each operation, the flow may perform a live transfer, place the caller in a human queue, create a ticket, or request a callback. What matters is that the conversation does not invent the result: the agent receives the real action status and tells the person only what the system confirmed.

Escalation separates three responsibilities: detecting the need for human help, executing an idempotent action, and communicating only the confirmed result.

Observing without entering the critical path

One of the most interesting parts of the design is voice-core. Its name might suggest that it processes the call, but it does not. It is a normalized observer that queries the public state of the voice plugin through RPC and turns changes into events such as:

• session.started
• participant.joined
• transcript.final
• playback.started
• session.ended

The observer periodically queries the state of calls, so it may reflect a transition with a small delay. That latency is acceptable for observability because it does not block the conversation: if the observer goes down, an active call can still be handled by the Gateway.

The conversation remains on the Gateway's critical path, while voice-core observes state through RPC and reconstructs the trace without blocking audio.

When the call ends, voice-core reconstructs the conversation and the tools persisted by the runtime. It can then export a logical call trace and retain the details needed to investigate a case: messages, executed tools, skills, escalation detections, and detected or confirmed fields.

Correlation uses a stable logical identifier per call, for example voice-call:<callId>. Physical identifiers that the runtime may rotate remain metadata. This prevents one conversation from being split into several traces just because the underlying technical sessions changed.

Two levels of tracing

To avoid mixing objectives, observability is divided into two levels:

1. Call trace: the telephone lifecycle, consolidated conversation, structured fields, skills, and results.
2. Per-turn agent traces: the prompt, response, model usage, context, and detailed tools.

In this implementation, Langfuse receives both levels grouped under the same logical session identifier. The first answers “what happened during this call?”. The second helps investigate “how did the agent respond during this turn?”.

There is an important limit: transcripts, phone numbers, contact reasons, and tool results may contain personal data. Tracing does not mean retaining everything without criteria. Access permissions, retention, and data usage are part of the security design, not a later analytics decision.

Conclusion

Building a phone agent is not a matter of putting TTS in front of a chatbot. It means designing an architecture in which real-time audio, the agent, security, and tracing have clearly separated responsibilities.

The modular STT → agent → TTS pipeline was a good starting point because it makes the controls explicit and allows each layer to evolve independently. An STS/realtime approach may be a valid improvement, but it must be evaluated as an architectural change, not as a simple provider selector.

Above all, a useful call is not one that merely responds naturally. It is one that can explain what it understood, what it validated, what action it actually executed, and what it left pending for a person.