VOOZH about

URL: https://dev.to/martschweiger/twilio-phone-agent-with-assemblyai-universal-3-pro-streaming-2nmp

⇱ Twilio Phone Agent with AssemblyAI Universal-3 Pro Streaming - DEV Community

Twilio Phone Agent with AssemblyAI Universal-3 Pro Streaming

Build an AI phone agent that handles real calls using Twilio Voice + Media Streams and the AssemblyAI Universal-3 Pro Streaming model for real-time speech-to-text.

The key detail: Twilio streams 8kHz μ-law audio. AssemblyAI Universal-3 Pro accepts pcm_mulaw at sample_rate=8000 natively — no resampling, no format conversion.

Architecture 

Incoming call
 │
 Twilio Voice
 │ TwiML → open WebSocket
 ▼
Your server (/media-stream WebSocket)
 │ │
 │ mulaw 8kHz audio │ synthesized mulaw audio
 ▼ ▲
AssemblyAI Universal-3 Pro ElevenLabs TTS
 │ transcript + turn signal
 ▼
 OpenAI GPT-4o

Prerequisites

Quick Start 

git clone https://github.com/kelseyefoster/voice-agent-twilio-universal-3-pro
cd voice-agent-twilio-universal-3-pro

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env

uvicorn server:app --host 0.0.0.0 --port 8000
ngrok http 8000

Configure Twilio

  1. Twilio Console > Phone Numbers > your number > Voice & Fax
  2. Set A Call Comes In to Webhook: https://your-ngrok-url.ngrok.io/incoming-call
  3. Call your Twilio number

AssemblyAI WebSocket Parameters for Twilio 

ASSEMBLYAI_WS_URL = (
 "wss://streaming.assemblyai.com/v3/ws"
 "?speech_model=u3-rt-pro"
 "&encoding=pcm_mulaw" # must match Twilio's audio format
 "&sample_rate=8000" # must match Twilio's 8kHz stream
 "&end_of_turn_confidence_threshold=0.5"
 "&min_turn_silence=400"
)

Phone calls have more background noise than browser audio. The higher confidence threshold and longer min_turn_silence reduce false triggers.

Post-Call Transcription 

import assemblyai as aai
transcriber = aai.Transcriber()
transcript = transcriber.transcribe(recording_url)
print(transcript.text)

Keyterm Prompting 

ASSEMBLYAI_WS_URL += "&keyterms_prompt=YourBrand&keyterms_prompt=SpecialTerm"

Deploy 

# Railway
railway login && railway init && railway up

# Render — Web Service with start command:
# uvicorn server:app --host 0.0.0.0 --port $PORT

Resources