[Platform] Introduce Speech support

[Guikingone]

Q	A
Bug fix?	no
New feature?	yes
Docs?	yes
Issues	--
License	MIT

• Introduce support for TTS, STT and STS for agents
• Add new configuration options to configure "speech" at agent-level.

Example for an OpenAI-based STS agent:

ai:
    platform:
        elevenlabs:
            api_key: '%env(ELEVEN_LABS_API_KEY)%'
        openai:
            api_key: '%env(OPENAI_API_KEY)%'

    agent:
        sts_openai:
            platform: ai.platform.openai
            model: gpt-4o
            speech:
                enabled: true
                platform: ai.platform.elevenlabs
                tts_model: eleven_multilingual_v2
                tts_options:
                    voice_id: some-voice-id
                stt_model: scribe_v1

• Agents can either be TTS or STT independently
• A SpeechConfiguration object handle the speech configuration
• A SpeechProcessor handle the input/output

[OskarStark]

To me we maybe should introduce capabilities also to platforms rather than having a voice component. As far as I understand I cannot use the Voice component standalone, right?

I don't think a dedicated component is the way to go here

[Guikingone]

We can introduce it via the Platform, could be easier, the voice can be used without agents but it will requires the Platform at least.

Will update the PR to match this approach 👍🏻

[OskarStark]

I agree, Agent scope is not needed 👍🏻

[chr-hertel]

Hi @Guikingone, i agree that week lack some kind of guidance on how voices work - but same goes for other binary stuff like creating images or videos.

so two things i would like to understand

• what's the high-level goal here - like what do you want to build?
• why is it an extra component and not part of Platform?

btw, "speech" is more common than "vioce" isn't it?
btw2, have you seen the demo around audio and video?

[Guikingone]

what's the high-level goal here - like what do you want to build?

The main goal is to add the capacity to have an agent/platform that can "listen" and answer to inputs thanks to voice / speech (voice is used as a sugar here, could be renamed to speech), creating a workflow where you can submit voice, call the platform that transforms it to speech / text (depending on the situation you're in) and returning it to the user without frictions.

why is it an extra component and not part of Platform?

It is now part of Platform, I just pushed an update on it following the comment from @OskarStark.

btw, "speech" is more common than "voice" isn't it?

Agreed, could be renamed to Speech.

btw2, have you seen the demo around audio and video?

Yes, the goal is to ease it with a "built-in" approach / API that stays transparent for the user.

[chr-hertel]

just realized we should the "audio" demo to "speech" as well - and i'm def not really happy with that solution there.

can we make it as easy as the structured output - like with an listener?

i like that starting point:

$result = $platform->invoke('eleven_multilingual_v2', new Text('Hello world'), [
    'voice' => 'Dslrhjl3ZpzrctukrQSN', // Brad (https://elevenlabs.io/app/voice-library?voiceId=Dslrhjl3ZpzrctukrQSN)
]);

echo $result->asVoice();

what would be the return type here? would it be same as asBinary() or asDataUri()

[Guikingone]

can we make it as easy as the structured output - like with an listener?

Could be something to explore, the API is not locked for now.

what would be the return type here? would it be same as asBinary() or asDataUri()

My first approach was to do the same thing as asBinary to ease the usage.

[Guikingone]

Well, might seems weird but here we go, stt, tts and sts are working like a charm ... 👀

[Guikingone]

Hi @chr-hertel / @OskarStark 👋🏻

Friendly ping on this PR, should I keep rebasing it / targeting 0.3 - 0.4 or should we rollback it to the Draft status?

[OskarStark]

@chr-hertel will have a look soon, not sure it will land in 0.3, lets keep it for now

[Guikingone]

Hi @OskarStark @chr-hertel, yes, I know, again 😄

I think that this time, that's the one, while thinking about #1572 and the comment from chris, I thought about this PR and the listener approach didn't looked like "THE" solution, especially while we have the processors, so, I asked Claude (yes, sometimes, asking for an external opinion might lead to a solution) for a "reworked implementation" that could ease the user experience and the maintenance of it, it submitted a solution close to the processors and I did the final tweaking.

So, what changed?

Now, the speech configuration is moved where it needs to be, at the Agent level, each agent can specify which platform to use, the options and so on, no more "in or out option", the agent handle the configuration, simple, clever, straight to the point.

The Platform still an important aspect of the PR (as the configuration is in the Platform but shared with the Agent), I also simplified the DI experience along with the code in the SpeechProcessor (the main entry point for speech now).

I updated the examples and reworked the documentation, much better, make more sense IMHO to be like that.

I let you take a look at it and review it if you think it deserves to be reviewed, is #1572 needed anymore? Thought question, if this PR is merged, probably not, at least, I don't see use case except for the validation/evaluation part (for now) that could require it (as speech is now at the agent level), probably another topic for another day 😄

[Guikingone]

A set of small improvements on the API, smaller scope for result support of Speech.

CI is failing due to vektor bridge, not tied to the PR itself.

[chr-hertel]

Can we reduce this to be just a decorator for an AgentInterface? like SpeakingAgent or SpeechAgent?
and document how to wire it as a service instead of extending the agent config in bundle?
would reduce the impact and effort here.

[Guikingone]

Could be a solution, the only issue it will not be "as smooth" as in the PR thanks to the configuration and the auto-injection of the processor, I'll take a look in that direction but can't promise anything this week.

[Guikingone]

Hi @chr-hertel, hope you're fine 👋🏻

I pushed the commit that contains the decorator approach, you were right about this approach, make sense as it removes the unnecessary configuration in agent section, I also updated the documentation to explain how to use it (I can add a extra section for the injection using Autowire if needed).

I let you review it when you have time 🙂

PS: If the PR is good for you, I'll fully squash it before you merge it, this way we get a clean history.

[chr-hertel]

Hey @Guikingone, yes i like - thanks for keeping this alive - i feel like we're on a good track.

Some things to tackle from my point of we:

• I think we should merge the additional methods of Speech into the DeferredResult and get rid off the extra data layer there, like Speech itself and that SpeechAwareInterface - i think it is basically only asDataUri missing there - would slim this down more and reducing the impact (e.g. not having SpeechAwareTrait in TextResult)
• Funnily enough @OskarStark and I just had a discussion yesterday about adopting Options (see notifier) or the OptionsResolver and decided against it since it is also limiting users - in this case i don't even know why we can't just have discrete properties in the SpeechConfiguration - it plays well with the extensibility of form component, but I don't see the same case here at this point
  => removing symfony/options-resolver in favor of well-defined SpeechConfiguration

Things I'm currenty unsure about:

• Extending agent's bundle config vs. standalone service registration - I'm slightly in favor of having it in the agent like you did, yes - not entirely sure tho
• Extending the MessageBag - that thing is growing - don't have a good alternative and see why you decided for it

My thoughts for now on rather high level - thanks again!

[Guikingone]

I'll take a look at points you highlighted, I can get rid of the OptionsResolver, not a big work on this side, I agree on the method for speech, could be merged in the result too.

Regarding the MessageBag, don't know, don't have a strong opinion on it, one solution might be to decorate it but for which use case? To do what and with which methods? 😅

For the configuration, my favorite approach would be to extend the existing one rather than depending on the DIC, I think that having it through service decoration is kind of an anti-pattern and can be easily simplified by the configuration, if you have multiple "speech agents", you'd be forced to declare them one by one while in the same time, keeping track of your agents in the configuration, looks like a "wrong turn" to take 🤔

[chr-hertel]

Alright, let's keep MessageBag and config like you changed already 👍

[Guikingone]

Here's the "final draft" using the decorator and the configuration, I also updated the demo to use the new SpeechAgent (it's fully transparent for the end user), the documentation contains the detailed process on how to use the speech support at both the Agent and AiBundle levels, tests also updated.

[Guikingone]

@chr-hertel I removed SpeechResult and updated the doc along with the examples, ready for review when you have time 🙂

[chr-hertel]

Thanks @Guikingone - will patch some inconsistencies across Cartesia, ElevenLabs and OpenAI TTS regarding support of plain string vs Text instances while merging - and my last comment about the text vs speech in metadata was confusing - will flip that as well.

Thanks again - great improvement! 👍

[Guikingone]

Oops, sorry, pushed the fix for the CHANGELOG.md at the same time 😅

[chr-hertel]

@Guikingone all good 😂 the risk of me force pushing into someone else's branches 😂

[chr-hertel]

Thank you @Guikingone.