How the speech pipeline works, why end-to-end latency under a second or two is what makes it feel human, how to design a call flow that actually handles your calls, and what to measure once it is live. Written so you understand the system before you ever hand us a phone number.
Automating phone calls used to mean press-one-for-billing IVR trees that callers hated. That changed. A modern AI voice agent holds a real, free-form conversation, checks your calendar, books the appointment, answers questions from your actual data, and hands off to a person when the call needs one. The technology is ready. The hard part is designing and deploying it well, which is what this guide covers in full.
Every call passes through three stages in a tight loop. Understanding them tells you why some agents feel natural and others feel slow.
Speech to text (STT). The caller speaks and a fast transcription model converts audio to text, typically within 200 to 400 milliseconds. The quality of this step matters a great deal in noisy environments, on accented speech, and on domain-specific vocabulary. A good system uses a model tuned for telephony audio, not one optimised for quiet studio recording.
A language model with tools. The transcript lands in a language model that has access to a set of tools: look up calendar availability, fetch a customer account, book an appointment, retrieve a price. The model decides which tools to call and in what order, then composes a spoken reply. This is where the intelligence lives, and it is also where most latency comes from if the model or the tool calls are slow.
Text to speech (TTS). The model's reply is converted back to audio and streamed to the caller, again within a few hundred milliseconds. Modern neural TTS at low latency sounds natural. The voice can be configured for gender, accent, pace, and warmth. It is not the clipped robotic voice of a decade ago.
The round trip through all three stages, plus any tool calls, is what callers experience as the agent's response time. At under one second, the conversation feels natural. At two seconds, callers notice a pause but accept it. Past three seconds, it feels broken. Hitting that target reliably is the main engineering challenge, and it is why infrastructure, model selection, and tool design all matter.
We open every engagement with a free pilot, because it is the only honest way to test whether the call flow works on your real calls. Here is what happens:
You tell us what calls you want handled. We design the call flow, configure the voice, and connect it to a test number. Within a few days you can call that number yourself and hear the agent in action. We record the call and share the transcript. You see the resolution, the latency, the handoff point. You do not commit to anything until you have heard it.
After the pilot, you keep the recording and the call-flow design whether or not you work with us further. A real call on a real flow is a useful artifact on its own: it ends the guessing that kills most voice AI business cases before they start.
The call flow is the logic the agent follows: what it handles itself, what it asks the caller, when it looks something up, and when it transfers. Getting this right is most of the work. A well-designed call flow with a mediocre model outperforms a brilliant model on a poorly designed flow, every time.
The design starts with a question: what are the actual calls coming in? Not what you think they are, but what they actually are. A week of real call logs almost always reveals that a small number of call types account for the majority of volume. Those are the ones worth automating first.
For each call type, we define two things: what the agent handles end to end, and what it transfers. The rule is not about complexity, it is about predictability. If the agent can answer reliably from its data, it handles it. If the answer might be wrong or the stakes are high, it transfers. The caller never hits a dead end.
A good call flow also has graceful recovery built in. If the caller says something unexpected, the agent acknowledges it and redirects rather than repeating the same prompt. If the agent cannot resolve the call, it offers to transfer or to take a message. There is always a path forward.
An agent that guesses is dangerous. If a caller asks whether a slot is available on Thursday at three in the afternoon and the agent says yes when the calendar is full, you have a problem. The fix is grounding: the agent pulls live data for every answer that depends on it.
What that means in practice:
Grounding adds a small amount of latency to each turn where a lookup is needed, typically 100 to 300 milliseconds. That is almost always worth it. An agent that never guesses earns caller trust and compounds in reliability over time.
The guessing trap. A language model asked a question it does not have data for will often produce a plausible sounding answer rather than admitting uncertainty. Grounding is the engineering practice that prevents this in production. We design grounding into every tool call from the start.
Real phone conversations do not follow a strict you-speak-then-I-speak pattern. Callers interrupt, confirm while the agent is still talking, or change direction mid-sentence. A voice agent that ignores this sounds robotic and frustrating. A well-built one handles it naturally.
Barge-in is the ability to detect that the caller has started speaking while the agent is still talking, and to stop the agent's audio immediately. Without it, the agent will finish its sentence over the caller, which most people find rude. With it, the agent pauses, listens, and responds to what the caller actually said.
Turn-taking detection is subtler. The agent needs to know when the caller has finished speaking, without waiting too long (which feels unresponsive) or cutting in too early (which feels pushy). The right threshold depends on the conversation context: a simple yes or no gets a short silence threshold; a caller thinking through a complex answer gets more space. Good turn-taking design is one of the details that separates agents that feel natural from ones that feel like a demo.
Filler handling also matters. When the agent is processing or calling a tool, it can produce a brief natural acknowledgment, "let me check that for you," rather than sitting in silence. A half-second of silence in a phone call feels long. A filler phrase fills it naturally and sets the right expectation.
Every call flow needs a human handoff path, and it needs to be designed to work well, not just to exist. A transfer that dumps the caller into a queue with no context is almost as bad as no transfer at all.
When the agent decides to transfer, it does two things. First, it tells the caller what is happening and roughly how long to expect: "I am connecting you with someone from the team now, they will be with you in a moment." Second, it passes a context packet to the human picking up: who the caller is, what they asked, what the agent already did, and why the transfer happened.
That context packet means the human picks up informed. They do not ask the caller to repeat their name or their problem. In a well-built system, the context appears in the agent's dashboard or is spoken to the human as a whisper before they connect. Either way, the caller experience is continuous, not fragmented.
Transfers happen on explicit rules: the caller asks to speak to a person, the agent has tried twice without resolution, the call type is flagged as always-human, or the confidence in an answer falls below a threshold. These rules are defined in the call flow and tuned during the pilot.
A modern AI voice agent can operate in any language with strong ASR and TTS coverage. Today that includes all major European languages (English, Italian, Spanish, French, German, Portuguese, Dutch), the main Asian languages (Japanese, Mandarin, Korean), and many others. Multi-language deployments are common when you serve callers across more than one region.
The agent can detect the caller's language automatically and switch mid-conversation if needed. Or it can be configured to operate in a single language on a specific number, with a separate number for each region. The right design depends on your call routing setup.
Voice selection is part of every build. We choose from a range of neural voices that fit your brand in terms of gender, accent, and tone, and we can configure pace and warmth. You listen to the voice on a pilot call before anything goes live. If it does not fit, we change it. The voice is not locked in.
The agent needs a phone number to answer, and that number needs to connect to the AI pipeline. There are a few ways to arrange this, and none of them require you to replace your existing phone system.
Twilio. The most common deployment path. Twilio provides a programmable phone number and a media stream the agent reads from and writes to. It handles the PSTN connection so you do not have to. We provision the number, configure the stream, and wire it to the agent.
SIP trunks. If you already have a VoIP system or PBX, the agent can sit in front of it via a SIP trunk. Calls arrive at the trunk, the agent handles them, and transfers go back through the trunk to your existing team. No hardware changes, no new carrier required.
Porting or pointing your existing number. You do not need a new number. You can port your existing business number to a Twilio account, or point it to the agent via call forwarding at your current carrier. Callers dial the same number they always have. They just reach the agent instead of an unanswered queue.
For businesses that run after-hours coverage, a common design is conditional call forwarding: calls during business hours go to your team, calls outside hours forward to the agent. This avoids any disruption to your live operation while capturing the after-hours volume that was previously being lost.
| Telephony path | When to use it | What changes at your end |
|---|---|---|
| Twilio-provisioned number | No existing number or starting fresh on a dedicated line | Nothing; we provision and configure the number |
| SIP trunk to your PBX | You have an existing VoIP or PBX system and want to keep it | A SIP trunk credential; no hardware change |
| Number porting | You want callers to reach the agent on your existing business number | Port request to Twilio; typically 2 to 4 weeks |
| Call forwarding at carrier | Quickest path; keep the number at your carrier and forward to the agent | A forwarding rule at your current carrier; same-day change |
| Conditional forwarding | Agent covers after-hours only; team takes business-hours calls | A time-based forwarding rule at your carrier or PBX |
The value of a voice agent compounds when it writes the result of every call back into the systems you already use. A call that ends with no record is a call that could have been handled better. A call that ends with a structured summary in your CRM is a call that improves your operations.
What lands in the CRM after every call:
Calendar booking is a first-class action in the call flow. The agent checks availability via your calendar API, books the slot while the caller is still on the line, and sends a confirmation. The booking is real, in real time. There is no lag, no manual step between the call and the appointment appearing in your schedule.
We integrate with the CRM and calendar systems you already run. The common ones are Salesforce, HubSpot, Zoho CRM, Google Calendar, Calendly, and any system that exposes an API. The integration is part of the build, not a separate project.
AI voice agents operate in a regulated space. Two requirements apply to almost every deployment, and several others apply depending on your jurisdiction and industry.
AI disclosure. In most jurisdictions, and certainly under EU AI Act provisions that came into force in 2025, an AI system engaging in real-time voice conversation must identify itself as such. We build this into the greeting by default. The agent says its name, identifies that it is an AI, and tells the caller who it represents. This is not a liability hedge, it is the right design: callers who know they are speaking to an AI cooperate better and are less frustrated when the agent has limits.
Recording consent. If the call is recorded, consent requirements vary by region. In the United States, some states require one-party consent and others require two-party consent. In the EU, GDPR makes explicit consent the safe default. We design the greeting and the consent flow for your specific operating regions, and we document the legal basis for recording in the engagement.
Other compliance areas worth naming: HIPAA applies if the agent handles health information in the US and governs what can be stored and where. PCI DSS applies if the agent collects payment card data over the phone, which generally means designing the call flow to avoid it. Sector-specific rules in financial services, legal, and other regulated industries layer on top of these. We scope compliance carefully before any deployment in a regulated context.
A voice agent that has no measurement is a voice agent you cannot improve. These are the numbers that matter, with honest targets based on real deployments:
| Metric | What it tells you | Realistic target range |
|---|---|---|
| Containment rate | Share of calls resolved by the agent without a human | 60% to 80% for well-scoped call types |
| Booking rate | Share of calls where the intended action (appointment, lead, callback) was completed | 70% to 90% of in-scope booking calls |
| Average handle time | Mean call duration from answer to resolution or transfer | 90 to 180 seconds for most routine call types |
| Missed-call recovery | After-hours calls captured that would otherwise have been lost | Close to 100% once live, since the agent always answers |
| Transfer rate | Share of calls transferred to a human; high transfer means the scope is too wide or the data is missing | Under 30% for a well-tuned deployment |
| Latency P95 | 95th percentile round-trip response time; callers feel this as naturalness | Under 1,500 ms for a good experience |
These targets are realistic, not aspirational. A deployment that starts below them and is well-instrumented will reach them within a few weeks of tuning. A deployment with no measurement will not improve at all.
The pilot gives you baseline numbers on your own calls before you commit to the full build. That baseline is the most honest foundation for projecting the ROI, because it is real data from your real call types, not an industry average applied to a situation it may not fit.
Voice agents that depend on external systems inherit the reliability of those systems. A calendar API that is down, a CRM that is slow, a tool call that times out: each of these can degrade or break the call if the agent is not designed to handle them.
Good fallback design looks like this: if a tool call fails, the agent acknowledges the issue in natural language and takes the next best action. If the calendar is unavailable, the agent takes the caller's preferred time and notes it for a human to confirm, rather than telling the caller it cannot help. If the CRM is slow, the agent proceeds with what it knows and queues the write for when the connection recovers.
At the infrastructure level, we deploy with redundant model providers and automatic failover on the STT and TTS steps, so a provider outage does not take the agent offline. Uptime targets for a production deployment should be in the high nineties. Anything lower is a sign that the failover design needs work.
Monitoring matters too. Every component of the pipeline should emit latency and error metrics, and there should be an alert when the containment rate drops unexpectedly or the P95 latency spikes. These signals catch problems before callers notice them.
The shape of the work is a free pilot, then a fixed-scope build, then optional ongoing operation. The pilot tests the call flow on your real calls before any money changes hands. The build wires it into your telephony, CRM, and calendar with proper monitoring and compliance. Operation means we continue tuning the agent as your call mix changes, adding new call types and keeping the performance metrics in range.
This is one of Scalarly's AI services. The same measure-first, build-a-real-workflow approach runs across the studio. If your operations involve large volumes of documents alongside calls, you may also want to look at Document AI, which applies the same engine to invoice processing, contract extraction, and certified digitization.
That is the complete engine. The next step is a free pilot on your own calls: a real call flow, a real voice, a real containment rate measured before you commit to anything.
A free call-flow pilot on your real calls, the containment rate measured, the bookings counted. Then we go live on your number.
Get a free pilot