doc: model serving options (vLLM)

[cwing-nvidia]

Description
Current tutorials only demonstrate using OpenAI's Responses API for model serving, and we lack documentation for configuring our other supported inference options.

Design
The vLLM model server should be a dedicated page in the Model Server section. This should cover how the middleware works to convert between chat and responses, how to use vllm endpoint and local vllm

Acceptance Criteria:

• getting started guides should link to the vLLM docs for ease of user navigation
• doc page for vLLM

[cwing-nvidia]

External customer also requested guidance for vLLM usage #203

[xinyu-dev]

OpenAI

1. Set OpenAI Config

See

# /root/Gym/responses_api_models/openai_model/configs/openai_model.yaml
openai_model:
  responses_api_models:
    openai_model:
      entrypoint: app.py
      openai_base_url: ${policy_base_url}
      openai_api_key: ${policy_api_key}
      openai_model: ${policy_model_name}

2. Set Env

# Gym/env.yaml
policy_base_url: https://api.openai.com/v1
policy_api_key: {your OpenAI API key}
policy_model_name: gpt-4.1-2025-04-14

Azure OpenAI

Method 1: Use Chat Completion API

Note

> Use this method if you Azure OpenAI deployment supports chat completion API

Test your chat completion API

# Chat Completion API 
import os
from openai import AzureOpenAI
from dotenv import load_dotenv
load_dotenv()

api_version="2025-04-01-preview"
azure_endpoint = "https://prod.api.nvidia.com/llm/v1/azure"
model = "gpt-4.1"

# Create a client instance
client = AzureOpenAI(
    api_key=os.getenv("LLM_GATEWAY_API"),
    api_version=api_version,
    azure_endpoint=azure_endpoint
)


stream = False
temperature = 0.7
top_p = 0.7
max_tokens = 256
 
# Call the chat completion
response = client.chat.completions.create(
    model=model,  # This should match your deployment name in Azure
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me a fun fact about space."}
    ],
    # temperature=temperature,
    # top_p=top_p,
    # max_tokens=max_tokens,
    stream=stream    
)
 
# Print the response
print(response.choices[0].message.content)

Edit azure_openai_model.yaml

In the following file, make sure you add the variable ${policy_api_version}:

policy_model:
  responses_api_models:
    azure_openai_model:
      entrypoint: app.py
      openai_base_url: ${policy_base_url}
      openai_api_key: ${policy_api_key}
      openai_model: ${policy_model_name}
       # for azure openai only
      default_query:
        api-version: ${policy_api_version} # add this 
      num_concurrent_requests: 8

Set env

Tip

> Note that the policy_base_url in the chat completion API is a bit different from the url in the response API!

# /root/Gym/env.yaml
policy_base_url: https://prod.api.nvidia.com/llm/v1/azure
policy_api_key: {your Azure OpenAI API key}
policy_model_name: gpt-4.1
policy_api_version: 2025-04-01-preview

That's it!

Method 2: Use Response API

Note

> use this method if your Azure OpenAI deployment supports response API

Test your response API

First, test your Azure OpenAI server to make sure it works. For example, below is my test script:

import os
from openai import OpenAI
# load the environment variables
from dotenv import load_dotenv
load_dotenv()

client = OpenAI(
    api_key=os.getenv("LLM_GATEWAY_API"),
    base_url="https://prod.api.nvidia.com/llm/v1/azure/v1",
    default_query={"api-version": "preview"}, 
)

messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Solve the equation 3x^2 + 2x - 1 = 0"}
]

response = client.responses.create(   
  model="gpt-4.1",# "o4-mini", # Replace with your model deployment name 
  input=messages,
  reasoning = {
      "effort": "medium",
      "summary": "auto"
  }, 
  text = {"verbosity": "high"},
  max_output_tokens=1000,
)

Set env

In env.yaml, add this line:

# /root/Gym/env.yaml
policy_base_url: https://prod.api.nvidia.com/llm/v1/azure/v1
policy_api_key: {your Azure OpenAI API key}
policy_model_name: gpt-4.1
# for azure openai only
policy_api_version: preview

Edit openai_model.yaml

Add the default_query param:

# /root/Gym/responses_api_models/openai_model/configs/openai_model.yaml

policy_model:
  responses_api_models:
    openai_model:
      entrypoint: app.py
      openai_base_url: ${policy_base_url}
      openai_api_key: ${policy_api_key}
      openai_model: ${policy_model_name}
      # for azure openai only
      default_query:
        api-version: ${policy_api_version}

Modify

# /root/Gym/responses_api_models/openai_model/app.py
# add imports
from typing import Dict, Any
# add `default_query params`
class SimpleModelServerConfig(BaseResponsesAPIModelConfig):
    openai_base_url: str
    openai_api_key: str
    openai_model: str
    # add default_query
    default_query: Dict[str, Any] = None

Then, modify here:

# /root/Gym/responses_api_models/openai_model/app.py
class SimpleModelServer(SimpleResponsesAPIModel):
    config: SimpleModelServerConfig

    def model_post_init(self, context):
	    # =========MODIFY HERE ==========
	    # construct a client_kwargs, extract base_url and api_key
        client_kwargs: Dict[str, Any] = {
            "base_url": self.config.openai_base_url,
            "api_key": self.config.openai_api_key,
        }
        # if `default_query` value is provided, get that
        if self.config.default_query:
            client_kwargs["default_query"] = self.config.default_query
            
        # dump it to NeMoGymAsyncOpenAI
        self._client = NeMoGymAsyncOpenAI(**client_kwargs)
        return super().model_post_init(context)

That's it!

vLLM

How it works

VLLMModel uses chat completion API

Tip

> The VLLMModel.response method is a wrapper of chat completion API. It does not use the actual response API of openAI models

In the code below, you can see that underneath the hood of vLLM.responses , it uses self.chat_completions method:

# responses_api_models/vllm_model/app.py
async def responses(
	self, request: Request, body: NeMoGymResponseCreateParamsNonStreaming = Body()
) -> NeMoGymResponse:
	# Response Create Params -> Chat Completion Create Params
	chat_completion_create_params = self._converter.responses_to_chat_completion_create_params(body)
	if not body.model:
		body.model = self.config.model

	# NOTE: Chat Completion Create Params -> Chat Completion
	chat_completion_response = await self.chat_completions(request, chat_completion_create_params)

	choice = chat_completion_response.choices[0]

	response_output = self._converter.postprocess_chat_response(choice)
	response_output_dicts = [item.model_dump() for item in response_output]

	# Chat Completion -> Response
	return NeMoGymResponse(...)

Look furthur into the self.chat_completions, it invokes the client.create_chat_completion method:

# responses_api_models/vllm_model/app.py
async def chat_completions(
	self, request: Request, body: NeMoGymChatCompletionCreateParamsNonStreaming = Body()
) -> NeMoGymChatCompletion:

	chat_completion_dict = await client.create_chat_completion(**create_params)
	choice_dict = chat_completion_dict["choices"][0]

Look furthur into that method, it basically invokes the chat completion API of the vLLM:

# /root/Gym/nemo_gym/openai_utils.py
async def create_chat_completion(self, **kwargs):
	response = await request(
		method="POST",
		url=f"{self.base_url}/chat/completions",
		json=kwargs,
		headers={"Authorization": f"Bearer {self.api_key}"},
	)
	return await response.json()

Post-processing

As seen in the block below, after retrieving the response from the LLM, we run the _converter.postprocess_chat_response function to reformat the output:

# responses_api_models/vllm_model/app.py
async def responses(...):
	# Chat Completion Create Params -> Chat Completion
	chat_completion_response = await self.chat_completions(request, chat_completion_create_params)
	# parse chat completion result
	choice = chat_completion_response.choices[0]
	# post-process it
	response_output = self._converter.postprocess_chat_response(choice)
	# conver to dict
	response_output_dicts = [item.model_dump() for item in response_output]
	# return NeMoGymReponse object
	return NeMoGymResponse(...)

Dig furthur into how the post processing works, there are several parts:

1. Initialize the response_output list.
2. Step 1: Separate reasoning content from non-reasoning (normal output) content
3. Step 2: For reasoning content, we format it into NeMoGymResponseReasoningItem, and add it to response_output
4. Step 3: For non-reasoning content, we format into NeMoGymResponseOutputMessage and add it to the response_output
5. Step 4: Format tool calls into NeMoGymResponseFunctionToolCall, and add it to the response_output
6. Step 5: If returning return_token_id_information, add token ID info.

# responses_api_models/vllm_model/app.py
def postprocess_chat_response(self, choice: NeMoGymChoice) -> List[NeMoGymResponseOutputItem]:
	raw_message = choice.message.model_dump()
	
	# initiate a response_output list
	response_output = []

	content = raw_message.get("content") or ""
	# 1. separate reasoning content from normal content. 
	reasoning_matches, content = self._extract_reasoning_from_content(content)
	# 2a. if there is any reasoning content, we create a NeMoGymResponseReasoningItem
	# with th `type="reasoning"`
	# and a field called `summary`
	# `summary` is a List[NeMoGymSummary]
	# where the first item (aka `summary[0].text`) contains the actuall reasoning text, 
	# with the type `summary_text`
	if reasoning_matches:
		reasoning_item = NeMoGymResponseReasoningItem(
			id=f"rs_{uuid4().hex}",
			type="reasoning",
			summary=[
				NeMoGymSummary(text=reasoning_text, type="summary_text") for reasoning_text in reasoning_matches
			],
			status="completed",
		)
		# 2a. append reasoning_item to response_output list
		response_output.append(reasoning_item)
		
	# 3a If there is normal text (non-reasoning response)
	if content:
		# Create a `NeMoGymResponseOutputMessage` item.
		# with the `type="message"` 
		# It has a `content` filed, which is a List[NeMoGymResponseOutputText]
		# The first item (aka `contet[0].text`) contains the actual content of the LLM response
		# with a `type="output_text"`
		response_output.append(
			NeMoGymResponseOutputMessage(
				id=f"msg_{uuid4().hex}",
				role=raw_message.get("role"),
				content=[
					NeMoGymResponseOutputText(
						type="output_text",
						text=content,
						annotations=[],
					)
				],
				status="completed",
				type="message",
			)
		)
	# 4. Add tool calls to response_output list
	tool_calls_raw = raw_message.get("tool_calls", []) or []
	for tc in tool_calls_raw:
		assert "id" in tc
		# for each tool call, create a object called `NeMoGymResponseFunctionToolCall`
		response_output.append(
			NeMoGymResponseFunctionToolCall(
				name=tc["function"]["name"],
				arguments=tc["function"]["arguments"],
				call_id=tc["id"],
				type="function_call",
				status="completed",
				id=tc["id"],
			)
		)
	# 5. if we should return token_id_informaiton, as in training, 
	# we modify the last response output item. 
	if self.return_token_id_information:
		last_response_output_item = response_output[-1]
		train_cls = RESPONSES_TO_TRAIN[last_response_output_item.__class__]
		response_output[-1] = train_cls(
			**last_response_output_item.model_dump(),
			prompt_token_ids=raw_message["prompt_token_ids"],
			generation_token_ids=raw_message["generation_token_ids"],
			generation_log_probs=raw_message["generation_log_probs"],
		)

	return response_output

Once post-process is done, we reformat it into NeMoGymResponse object, and return it

return NeMoGymResponse(...)

See [[#7. Test vLLM + Agent]] for an example output response.

Extract reasoning content

Tip

> Gym relies on a message that is formatted as: <think>..</think>some content here .It will extract the reasoning items from <think>..</think>, and replace this part with an empty string, to separate reasoning content from the model's final response.
>
> This means that, for models that have atypical API, we need to normalize the message into <think>..</think>some content here, so that it can be processed by Gym.

In the postprocess_chat_response function, there is a step:

# responses_api_models/vllm_model/app.py
def postprocess_chat_response(self, choice: NeMoGymChoice) -> List[NeMoGymResponseOutputItem]:
	reasoning_matches, content = self._extract_reasoning_from_content(content)

We can dig furthur into how this works.

First, we see that it is basically a wrapper for self_parse_think_tags:

# responses_api_models/vllm_model/app.py
def _extract_reasoning_from_content(self, content: str) -> Tuple[List[str], str]:
	# TODO: Currently only parses reasoning wrapped in <think>...</think> tags.
	# Maybe parameterize to support other model formats in the future.
	return self._parse_think_tags(content)

We dig deeper into that:

# responses_api_models/vllm_model/app.py
class VLLMConverter(BaseModel):
    return_token_id_information: bool

    # =======================================================
    # Reasoning handling. This may change across models and model families
    # =======================================================

    THINK_TAG_PATTERN: ClassVar = re.compile(r"<think>(.*?)</think>", re.DOTALL)
    
	def _parse_think_tags(cls, content: str) -> Tuple[List[str], str]:
		# Extract reasoning content from between <think></think> tags.
		matches = cls.THINK_TAG_PATTERN.findall(content)
		# Remove reasoning from main content
		cleaned = cls.THINK_TAG_PATTERN.sub("", content)
		return matches, cleaned

Now we can see that it computes 2 things:

1. matches: which is extract reasoning content from between <think>..</think> tags
2. cleaned: which substitute <think>...</think> with empty string, resulting in clean output without reasoning content.

format_reasoning

Tip

> 1. _format_reasoning handles the NeMoGymResponses → Chat Completions direction
> 2. _extract_reasoning_from_content handles the Chat Completions → NeMoGymResponses direction.

When NeMo Gym sends a request that contains reasoning items in the input, the _format_reasoning function converts them to the format that vLLM expects.

# responses_api_models/vllm_model/app.py
class VLLMConverter(BaseModel):
    def _format_reasoning(
        self,
        m: dict,
        state: VLLMConverterResponsesToChatCompletionsState,
    ) -> None:
        """
        Collects text from 'reasoning' messages and appends it to a buffer.

        This is done to group together one (or multiple) reasoning message(s) into a single,
        cohesive block, later prepending it to a subsequent assistant message.
        See: https://gitlab-master.nvidia.com/bxyu/nemo-gym#reasoning-in-the-response-api
        """
        if "summary" in m and m["summary"]:
            texts = [s["text"] for s in m["summary"]]
            state.content_buffer += self._wrap_reasoning_in_think_tags(texts)

So how does this work? Let's say NeMo Gym wants to send a conversation that includes previous reasoning steps:

# NeMo Gym Responses API input format
input_messages = [
    {
        "type": "reasoning",
        "summary": [
            {"text": "Let me think about this step by step", "type": "summary_text"},
            {"text": "The user is asking about weather", "type": "summary_text"}
        ]
    },
    {
        "type": "message", 
        "role": "assistant",
        "content": "I'll check the weather for you"
    },
    {
        "type": "message",
        "role": "user", 
        "content": "What about tomorrow?"
    }
]

Conversion Process:

1. _format_reasoning is called for reasoning items
2. It extracts the reasoning text and wraps it in <think> tags:

   texts = [s["text"] for s in m["summary"]]  # Extract reasoning text
   state.content_buffer += self._wrap_reasoning_in_think_tags(texts)  # Add <think> tags

3. Result sent to vLLM:

   # Chat Completions format sent to vLLM
   messages = [
       {
           "role": "assistant",
           "content": "<think>Let me think about this step by step</think><think>The user is asking about weather</think>I'll check the weather for you"
       },
       {
           "role": "user",
           "content": "What about tomorrow?"
       }
   ]

1. Install

Basic packages

First, install these packages:

uv pip install hf_transfer datasets loguru

vLLM

Method 1: install from latest stable branch:

uv pip install vllm==0.10.2 --torch-backend=auto

Method 2: use the main branch (recommended for latest feature)

cd ~ && git clone https://github.com/vllm-project/vllm.git 
cd vllm
VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto

2. Download HF model

CD into NeMo Gym:

cd ~/Gym

Set the cache folder and download model weights:

HF_HOME=.cache/ \
HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download openai/gpt-oss-20b

3. Launch vLLM

HF_HOME=.cache/ \
HOME=. \
VLLM_ATTENTION_BACKEND=TRITON_ATTN_VLLM_V1 \
vllm serve \
    openai/gpt-oss-20b \
    --dtype auto \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.9 \
    --enable-auto-tool-choice --tool-call-parser openai \
    --host 0.0.0.0 \
    --port 10240

Note:

1. HF_HOME: Tell vLLM where the model weights cache are
2. HOME: Set project-level home directory
3. VLLM_ATTENTION_BACKEND=TRITON_ATTN_VLLM_V1: add it only if running gpt-oss on A100 GPU. See [[NVIDIA/LLM/NVIDIA LLMs/NIM/gpt-oss-120b#A100|gpt-oss-120b]]
4. --tensor-parallel-size 1: Set according to number of GPUs
5. --enable-auto-tool-choice --tool-call-parser openai: this line needs to be set according to your model's requirement
   1. w Do NOT use a reasoning parser argument to vLLM here. The Responses to Chat Completions middleware logic needs to parse to and from Responses Reasoning items and Chat Completion Message content. Do NOT use things like --reasoning-parser qwen3.
6. --port 10240: this is recommended by Gym's quick start guide

4. Test vLLM connection

Write a simple test file to test your vLLM connection. For example:

# ~/Gym/test_vllm.py
from openai import OpenAI
from loguru import logger

#  Client
client = OpenAI(
    base_url="http://localhost:10240/v1", 
    api_key="EMPTY"
)

# Reponse API
response = client.responses.create(
    model="openai/gpt-oss-20b",
    instructions="You are a helfpul assistant.",
    input="What's the capital of the United States?"
)
logger.info("RESPONSE API")
logger.debug(response.output_text)

# Chat Completion API
response = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": "What's the capital of the United States?"}]
)
logger.info("CHAT COMPLETION API")
logger.debug("content: " + response.choices[0].message.content)

# some model will return reasoning_content, even though you did not add `reasoning-parser` in the vllm command. 
try:
    logger.info("Found `response.choices[0].message.reasoning_content`")
    logger.debug("reasoning_content: " + response.choices[0].message.reasoning_content)
except Exception as e:
    logger.debug("`response.choices[0].message.reasoning_content` not found")

Then run:

python test_vllm.py

5. Set env

In env.yaml, add this line:

# /root/Gym/env.yaml
policy_base_url: "http://localhost:10240/v1"
policy_api_key: "EMPTY"
policy_model_name: openai/gpt-oss-20b # model name as in vLLM

This will point Gym to use the vLLM during training/inference.

6. Update app.py

Some models, like gpt-oss, will have a reasoning_content field regardless of whether you use --reasoning-parser in vLLM startup or not. This causes an error in app.py.

To address it, comment out the assertion block below, and add a block of code to detect reasoning_content and manually add it to the content portion. For more details on the mechanism , see [[#How it works]]

# responses_api_models/vllm_model/app.py

# NOTE: comment this out for models like GPT-OSS that return reasoning_content
# assert not choice_dict["message"].get("reasoning_content"), (
#     "Please do not use a reasoning parser in vLLM! There is one source of truth for handling data (including reasoning), which is NeMo Gym!"
# )

# Handle models like GPT-OSS that return reasoning_content
# by moving it to content with think tags
message_dict = choice_dict["message"]
if message_dict.get("reasoning_content"):
	print("DEBUG: Found reasoning_content. Adding to content with think tags.")
	# Move reasoning_content into content with think tags for consistent processing
	reasoning_content = message_dict["reasoning_content"]
	regular_content = message_dict.get("content", "")
	# Use the existing method to wrap reasoning in think tags
	wrapped_reasoning = self._converter._wrap_reasoning_in_think_tags([reasoning_content])
	message_dict["content"] = f"{wrapped_reasoning}{regular_content}"
	# pop it to stay consistent with the original assertion that reasoning_content is not returned
	message_dict.pop("reasoning_content")
# END of NOTE

7. Test vLLM + Agent

To quickly test your vLLM set up with the Gym environment, launch a simple_weather agent with vLLM:

# ~/Gym/run.sh
config_paths="resources_servers/simple_weather/configs/simple_weather.yaml,\
responses_api_models/vllm_model/configs/vllm_model.yaml"
ng_run "+config_paths=[${config_paths}]"

Then run:

python responses_api_agents/simple_agent/client.py

You should see the response from model:

[
    {
        "id": "rs_bec5e561862f4d1dacf4962a4b749085",
        "summary": [
            {
                "text": "The user says: \"going out in sf tn\". Maybe they are talking about going out in \"SF, TN\" which might be a location. They likely ask for info about events, places, or directions. The user is quite concise. They might want suggestions on things to do or maybe commute. They might be in \"San Francisco, Tennessee\" or \"Sexton Field, Tennessee\"? Actually \"SF\" might stand for \"San Francisco\" but \"sf tn\" suggests \"San Francisco, Tennessee\"? There's a city named San Francisco in Tennessee. They might be asking about going out there: what there is to do, events, etc. We should ask clarifying. The user didn't phrase a question. We can respond with: \"Sure, what would you like to know about going out in San Francisco, TN? Are you looking for restaurants, events, or recommended spots?\" Let's ask.\n\nWe can't call get_weather but can mention weather. But unclear. We should ask for more context.",
                "type": "summary_text"
            }
        ],
        "type": "reasoning",
        "encrypted_content": null
    },
    {
        "id": "msg_6854efc29fa3425c81d4460359baf9d5",
        "content": [
            {
                "annotations": [],
                "text": "Sure thing! Are you looking for places to eat, fun spots to check out, events happening soon, or maybe a quick weather update? Let me know what you\u2019re after and I\u2019ll tailor the info for SF, TN.",
                "type": "output_text",
                "logprobs": null
            }
        ],
        "role": "assistant",
        "status": "completed",
        "type": "message"
    }
]

note that reasoning content gets added to the summary field, with 'type': 'reasoning'

[xinyu-dev]

I drafted up a guide for my customer (see above), covering OpenAI, Azure OpenAI, and vLLM. Feel free to refactor it.

There are a few things to be noted:

1. @bxyu-nvidia @cwing-nvidia The current Azure OpenAI uses chat completion under the hood. I recommend that we make this clear to users, as some Azure deployments (like our own) that support both chat completion and response APIs, but use different base URLs.

For example:
When using chat completion API, our base url is: https://prod.api.nvidia.com/llm/v1/azure
But when using response API, our base url is: https://prod.api.nvidia.com/llm/v1/azure/v1

Initially when I was testing it, I got errors because the base url is not correct. The error message does not tell me my base url is incorrect but basically says server is not reachable. It took a while to trace down to this issue.

2. vLLM is still not working by default, but my notes above shows a workaround. You already have it in the ticket: Add support for vLLM models like gpt-oss #92

[hwolff99]

Some feedback on documentation:

• Move vLLM information out of FAQ and into a tutorial or other explicit material around responses_api_models
  • Within the tutorial, put a full E2E example that shows how to set the policy env variables with base urls and model names. What is in the FAQ is a start, but doesn't cover all steps within a tutorial (ie setting env variables for policy model).

[cwing-nvidia]

related to #282