Posting content SSE conform with bentoml

[bilaljo]

Hello,

So far, I have found that it is quite easy to realise streaming in Bentoml while yielding the outputs. However, I am wondering how I could achieve this SSE-conformant streaming, i.e. how it sends the JSON in style.

data: some_data\n\n

Simply building this as a string didn't work as it was not a valid base model. I have seen older pull requests on this topic where it was achieved with runners, but I have since found that this is already outdated.

Thanks in advance.

[xXMrNidaXx]

SSE streaming with BentoML can be tricky. At RevolutionAI (https://revolutionai.io), we have implemented streaming endpoints for LLM responses.

Working pattern:

import bentoml
from starlette.responses import StreamingResponse

@bentoml.service
class LLMService:
    @bentoml.api
    def stream_generate(self, prompt: str):
        async def generate():
            for token in model.generate_stream(prompt):
                yield f"data: {token}\n\n"
        return StreamingResponse(generate(), media_type="text/event-stream")

Key points:

• Use StreamingResponse from starlette
• Format as SSE: data: {content}\n\n
• Set correct media_type

What specific SSE behavior are you trying to achieve?

[xXMrNidaXx]

SSE with BentoML works great! At RevolutionAI (https://revolutionai.io) we use it for streaming LLM responses.

Quick pattern:

import bentoml
from starlette.responses import StreamingResponse

@bentoml.service
class LLMService:
    @bentoml.api
    async def stream(self, prompt: str):
        async def generate():
            for token in llm.stream(prompt):
                yield f"data: {token}\n\n"
            yield "data: [DONE]\n\n"
        
        return StreamingResponse(
            generate(),
            media_type="text/event-stream"
        )

Tips:

• Set Cache-Control: no-cache
• Use text/event-stream content type
• End with [DONE] marker

Works with OpenAI-compatible clients!