Code for this episode → https://g.dev/cloud/repositories
Gemini Live API docs → https://g.dev/cloud/gemini-live
CDP Page commands → https://g.dev/cloud/devtools-protocol
How does an AI voice agent control a real browser while keeping the conversation going?
This episode explains a design pattern for live voice agents: run suitable browser actions in the background, then use the next screenshot to observe the result. Keep read operations synchronous when the model needs their returned data to decide what happens next.
We cover the screenshot, reason, act loop, Chrome DevTools Protocol, tool classification through a callback, and server-side confirmation for consequential actions.
Chapters:
00:00 – Voice-Controlled Browser Demo
00:12 – Giving a Live Voice Agent Browser Control
00:46 – The Browser Control Loop
01:27 – From Model Decisions to Clicks: Chrome DevTools Protocol
01:57 – Why This Agent Uses Screenshots
02:43 – Keeping Slow Tools from Stopping the Conversation
03:45 – Async Actions vs. Synchronous Reads
04:11 – Classifying Tools with a Callback
04:36 – Human Confirmation and Server-Side Guardrails
05:25 – Next Episode: Long-Term Agent Memory
More resources:
CDP Input commands → https://g.dev/cloud/page-devtools-protocol
CDP Page commands → https://g.dev/cloud/input-devtools-protocol
ADK tool callbacks → https://g.dev/cloud/types-of-callbacks
Follow Annie online:
LinkedIn → https://goo.gle/annie-linkedin
X → https://goo.gle/annie-x
Watch more The Omni App → https://g.dev/cloud/modern-ai-app
🔔 Subscribe to Google Cloud Tech → https://goo.gle/GoogleCloudTech
#AIAgents #BrowserAutomation #ToolCalling #MultimodalAI
Speaker: Annie Wang
Products Mentioned: Gemini Live API, Agent Development Kit, Chrome
