How AI Voice Agents Work: Speech-to-Speech or a Pipeline?

Most conversations about AI voice agents begin with the voice: how human it sounds, how quickly it answers. Those things matter, but they are the visible tip of a system with several moving parts. Understanding the parts helps you ask better questions of any supplier, including us, and avoid being sold a demo that falls apart on a real phone line.

This article is about how the pieces fit together. For what it costs, see how much an AI voice agent costs in the UK. For whether it suits a service business at all, read our service-business guide to voice agents.

The anatomy of a call

Whatever the architecture, the same journey happens.

1. 01TelephonyThe call arrives from a phone number via a carrier.
2. 02Audio transportCall audio is streamed to the voice service.
3. 03Voice modelSpeech is understood and a reply is produced.
4. 04ToolsYour systems are queried or updated through controlled calls.
5. 05OutcomeA booking, message or handover is completed and logged.

The architecture choice mostly changes the third step. The business value, however, tends to sit in the fourth and fifth: whether the agent can actually check a diary, create a record or pass the call to a person with context.

Three architectures compared

OpenAI's voice agents guide, checked on 10 October 2026, lists three architectures. Other vendors offer variations, and the trade-offs below are drawn from the vendor documentation rather than from our own benchmarks.

Three voice agent architectures and when each tends to fit

Architecture	How it works	Strengths	Trade-offs
Speech-to-speech (for example the Realtime API)	One model interprets audio, decides what to do and replies in speech within one session.	Voice-to-voice interaction without separate transcription and synthesis steps, which the vendor says lowers latency and keeps tone and inflection available to the model.	Less of the conversation passes through plain text you can inspect or rewrite before it is spoken. Control comes from prompts, tools and configuration.
Full-duplex with a separate backend (for example GPT-Live)	A live model handles the spoken exchange and delegates reasoning and tool work to a backend you choose.	Lets you keep an existing text workflow and its business logic, and the caller can keep talking while backend work runs.	Two systems to design and test: speaking behaviour in the live model, business rules in the backend.
Chained pipeline	Speech-to-text, then a text agent, then text-to-speech, managed by your application.	Each stage is visible and replaceable. You can store the transcript, run policy checks and speak only an approved answer.	More stages to manage, and each adds delay and a place for errors. Natural interruption handling takes more work.

xAI's Speech to Speech API follows the first pattern. It offers real-time voice over a WebSocket connection with server-side turn detection and tool use, including custom functions and remote tools. Other vendors offer variations on these patterns. The right choice for you is rarely about which has the nicest voice. It is about how much control you need over what is said.

Choosing by the control you need

A useful rule of thumb is to ask what would happen if the agent said something wrong.

• Low stakes (taking a message, answering opening hours): speech-to-speech is usually a good fit and the most natural to listen to.
• Medium stakes (booking into a diary, taking a deposit): either works, provided every action runs through a tool your own systems control and verify.
• High stakes (regulated wording, financial or medical statements, anything with legal consequences): a chained pipeline, or a design with human review, gives you the chance to check text before it is spoken. OpenAI's guide gives exactly this example: store the transcript, run policy checks, then generate speech once the workflow reaches an approved answer.

OpenAI's Realtime documentation also describes keeping voice activity detection while disabling automatic responses, so your own code decides when a response is created. It says this can suit moderation, input validation or retrieval patterns where you accept a little more latency for control.

Connecting to the phone network

A voice model is not a phone line. You need a route from the public telephone network to the model.

• Telephony providers such as Twilio can stream call audio over WebSockets. xAI publishes a cookbook example in which Twilio streams audio to the API and back, using native telephony audio formats to avoid conversion.
• SIP lets you route calls from a carrier or office phone system into a voice session. In xAI's documentation, a signed webhook announces each incoming call, and your server joins it over a WebSocket. The documentation tells you to verify the webhook's signature headers before trusting it.
• Browsers and apps typically use WebRTC. OpenAI recommends WebRTC rather than WebSockets for playing audio back on client devices because it copes better with uncertain networks.
• Transfers. xAI's SIP interface offers a call transfer request, and documents that a failed transfer leaves the caller connected. Whatever platform you use, design the handover path before launch.

Session limits exist too. OpenAI's Realtime documentation gives a maximum session duration of 60 minutes, which is rarely a problem for a phone call but is worth knowing.

Tools: where the agent touches your business

The model decides; your code acts. The documentation for both OpenAI and xAI describes the same loop for custom functions: the agent decides to call a function, your code runs it, you return the result, and the agent continues. A tool definition looks like this. It is an illustrative shape, so check your provider's documentation for the exact format.

{
  "type": "function",
  "name": "check_appointment_slots",
  "description": "Return available appointment slots for a date. Read-only.",
  "parameters": {
    "type": "object",
    "properties": {
      "date": { "type": "string", "description": "ISO date, for example 2026-11-02" }
    },
    "required": ["date"]
  }
}

Three design habits make a difference.

1. Start with read-only tools. Checking a diary is safer than writing to one. Add write actions once you trust the behaviour.
2. Keep permissions in your system, not the prompt. OpenAI's guide states that in its full-duplex designs your application controls permissions and business records. A prompt that says "never refund more than £50" is a request. A limit enforced in your API is a rule.
3. Confirm before committing. Have the agent read back the booking and ask for confirmation, then verify the saved record matches what was said.

This is the same discipline we describe when designing durable API integrations.

Testing a voice agent properly

OpenAI's guide is direct about this: test both the conversation and the completed task. For a booking assistant, listen to the confirmation and check that the correct appointment was saved. Its suggested progression is worth borrowing:

1. Crawl: synthetic speech and controlled single-turn requests, with the expected outcome fixed.
2. Walk: replayed recordings of real people, to expose accents, microphones and background noise.
3. Run: an independent simulated caller for multi-turn conversations, including interruptions and changes of mind.

Track latency using the median and the 95th percentile, not the average, and keep the caller, recording, prompt and transport constant when comparing changes. Add human listening on top, because pace and naturalness are hard to score automatically. If you need help building that safety net, our AI voice agents work includes it.

Security and privacy design

• Keep API keys on the server. xAI's reference says to use your key server-side only, or an ephemeral client secret for browser clients. Never ship a long-lived key in a web page or mobile app.
• Verify incoming webhooks. Check signatures and timestamps before acting on a call event.
• Treat what callers say as untrusted input. A caller can say anything, including instructions aimed at the model. Our prompt injection guide explains why permissions belong in your systems.
• Decide what you record and for how long. Recording is optional, and callers should be told. Our guide to telling callers it is an AI covers the EU and UK rules.
• Plan the human fallback. Know what happens when the model fails, the connection drops or the caller asks for a person.

Keeping options open

Voice models and prices are changing quickly. Keep your business logic behind your own API so that the voice layer can be swapped without rebuilding your integrations, and log enough to compare providers on your own calls. A voice layer that you cannot replace is a dependency you will pay for later. Our AI automation work is built on this principle.

Conclusion

Choosing between speech-to-speech and a pipeline matters, but less than people expect. What decides whether a voice agent is useful is the quality of its tools, the controls around them, and whether it was tested on realistic calls. Choose the architecture that gives you the control your risk level demands, build read-only first, and measure outcomes. If you want to talk through a design for your business, contact us about a voice agent.