[AI] Real-Time Abstractions

[jeffhandley]

We need abstraction APIs that support real-time streaming interactions across a variety of modalities. Some scenarios support one-way streaming and others support bidirectional streaming.

• Collect data about which providers support streaming requests and/or responses
• Define which modalities have real-time support across those providers
• Start with an app created using the AI Chat Web project template and integrate real-time interactions into the application with each of the providers/modalities
• Refactor out APIs for the abstractions
• Create the concrete implementations for those providers based on the abstractions
• Author samples/docs that demonstrate how to integrate real-time capabilities into an AI Chat Web app
• Consider integrating real-time interactions into the AI Chat Web project template

References

• Realtime API - OpenAI API
• Get started with Live API  |  Gemini API  |  Google AI for Developers

[jeffhandley]

There are some real-time speech-only services that could be implemented to test out the design. Amazon might have offerings as well. Goal is to have a project plan put together in November.

[tarekgh]

• Realtime Client Proposal #7285