Skip to content

Add configurable per-request LLM token and prompt-byte limits - #1026

Open
SajjadPSavoji wants to merge 1 commit into
NVIDIA:mainfrom
SajjadPSavoji:fix-1003-allow-configurable-per-request
Open

SajjadPSavoji wants to merge 1 commit into
NVIDIA:mainfrom
SajjadPSavoji:fix-1003-allow-configurable-per-request

Conversation

@SajjadPSavoji

Copy link
Copy Markdown

Deployments with a gateway that rejects big requests had to patch the code to get a smaller per-request size. This adds three optional env vars: SKILLSPECTOR_MAX_BATCH_INPUT_TOKENS, SKILLSPECTOR_MAX_PROMPT_BYTES and SKILLSPECTOR_TP4_MAX_BATCH_INPUT_TOKENS. Each one only lowers a limit, so the model budget and the CLI provider byte cap still win when they're smaller. Unset or blank keeps today's behavior and bad values log a warning without echoing the value.

The byte cap goes through the existing chunking in get_batches and TP4's own chunk planner. Prompts are also checked right before sending in the sync and async paths. A line or fixed prompt that can't fit is marked SIZE_LIMIT for that batch, nothing is truncated and other batches keep running. README has a short section on units and precedence.

Thanks @SajjadPSavoji for the suggested design in the thread.

Tests: new TestConfigurableRequestLimits in tests/nodes/test_llm_analyzer_base.py plus the TP4 token ceiling, byte ceiling and SIZE_LIMIT partial tests in tests/test_mcp_tool_poisoning.py. They fail before the change and pass after. I also ran the rest of test_llm_analyzer_base.py, test_mcp_tool_poisoning.py, test_meta_analyzer.py and test_meta_analyzer_fallback.py, and ruff check on the touched files.

Fixes #1003

Signed-off-by: Sajjad Pakdamansavoji <44399382+SajjadPSavoji@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow configurable per-request LLM token and prompt-byte limits for constrained deployments

1 participant