Source item
Open original source ↗Quoted context
These are the quoted author’s words.
For the past few months the @livekit team has been working with OpenAI to give developers access to the same technology that powers Advanced Voice in ChatGPT. With the new Realtime API, you can build voice AI that understands the nuances of human speech and responds in 300ms with humanlike expression. Under the hood, the Realtime API is built on websocket, which works well for server-to-server communication. But for delivering voice data to client devices, websocket isn’t the best choice. It’s not resilient to network congestion and wasn’t designed for transporting media in real time. This week, we released the Multimodal Agent API in our Agents framework. It completely wraps OpenAI’s Realtime API, abstracts away the raw wire protocol, and provides an ultra-low latency WebRTC transport between GPT-4o and your users’ devices We also released: Hooks, components, and visualizers for building voice AI frontends on web and mobile. “AI needs UI” — @AndrewYNg AgentsJS — build voice AI applications in Node environments: https://t.co/IYj1yr6ckE A Realtime API playground: https://t.co/c6hA9XDQpk We wrote some words too: Build your first app with the Realtime API: https://t.co/hMRrdsRmos Blog post with more details: https://t.co/t1jdWkaRPr We’re going to see a totally new class of voice AI applications get built with this API. I’d love to hear about what y’all use this API for and help in any way. DMs open!
dsa ↗