Streaming

Updated

Streaming returns output as the model generates it, which suits long answers and interfaces that want the first token fast.

Protocol How to enable End marker
OpenAI-compatible Add "stream": true to the body data: [DONE]
Anthropic-compatible Add "stream": true to the body The message_stop event
Gemini-compatible Call :streamGenerateContent?alt=sse End of stream; without alt=sse you get a JSON array 
curl -N https://modelportal.ai/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "MODEL_ID", "stream": true, "stream_options": {"include_usage": true}, "messages": [{"role": "user", "content": "Write a short poem"}]}'

from openai import OpenAI

client = OpenAI(base_url="https://modelportal.ai/v1", api_key="YOUR_API_KEY")
stream = client.chat.completions.create(
    model="MODEL_ID",
    messages=[{"role": "user", "content": "Write a short poem"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Usage

  • In the OpenAI format, add "stream_options": {"include_usage": true} to get usage in the last chunk; the platform does not add it to requests that did not ask.
  • Usage in the stream is the billed usage and matches your bill. The last event with usage also carries provider, attempts and cost in the usage object (the charge, in USD); every OpenAI Chat chunk carries provider. See the API overview.

Timeouts and interruptions

  • The first content event may take up to 600 seconds, shared by all attempts. Once output starts, a gap of more than 120 seconds between events closes the stream, and one output can run for at most 60 minutes.
  • While waiting for the first content event, the platform sends an SSE comment : keepalive every 15 seconds so that clients and proxies do not drop the idle connection. SSE clients ignore comments; in the Gemini JSON array (without alt=sse) the keepalive is a newline.
  • Headers go out when the first content event arrives or keepalives start, and always include X-ModelPortal-Request-Id.
  • A failure before the headers go out returns an ordinary HTTP error. A failure after that, including when every attempt fails after only keepalives were sent, ends the stream with an SSE error event. An interruption after output starts is billed for the usage produced so far; when exact usage is unavailable, an estimate is billed and marked Estimated on your bill.