introduce deterministic mode for reproducible LLM-based question generation#666
introduce deterministic mode for reproducible LLM-based question generation#666piyush06singhal wants to merge 2 commits intoAOSSIE-Org:mainfrom
Conversation
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 44 minutes and 37 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR adds optional deterministic question generation to the LLM question generator and corresponding backend endpoints. A new Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@backend/Generator/llm_generator.py`:
- Around line 50-52: Replace uses of Python's built-in hash(...) for seed
computation with a stable, cross-process hash: when computing seed_value (the
variable set when deterministic is true, e.g., the "Compute seed BEFORE text
truncation" block and the other seed computations around the same logic at the
other locations), generate a SHA-256 (or similar cryptographic) digest of the
input_text bytes and convert it to an integer, then take mod 2**32 for the seed;
keep the deterministic flag and None behavior unchanged. Locate the seed_value
assignment and any other places that currently use hash(input_text) and swap in
this stable hashing approach so identical input_text produces the same seed
across process restarts.
In `@backend/server.py`:
- Line 104: The deterministic assignment may treat string values like "false" as
truthy; add a small helper function (e.g., parse_bool(value, default=False)) in
server.py that accepts values of types bool, str, int and returns a strict
boolean (accept "true"/"1" case-insensitively as True, "false"/"0" as False, and
fall back to default), then replace the four occurrences that set deterministic
= data.get("deterministic", False) with deterministic =
parse_bool(data.get("deterministic", None), False) so each endpoint uses the
normalized boolean; ensure the helper is reused by all endpoints that reference
deterministic.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 5ecabd62-6862-44e9-9daf-5f869a87ee2c
📒 Files selected for processing (2)
backend/Generator/llm_generator.pybackend/server.py
1 participant
Summary
This PR introduces a deterministic mode for LLM-based question generation to enable reproducible outputs for identical inputs.
Currently, LLM-generated responses are non-deterministic due to stochastic sampling, which makes debugging, testing, and result comparison difficult. This change adds an optional mechanism to produce consistent outputs when required.
What was implemented
deterministicflag in LLM-based API endpointsllm_generator.pyto conditionally control generation parameterstemperatureis set to0.0top_pis set to1.0seedis derived from input text using:hash(input_text) % (2**32)temperature = 0.7)How this improves the system
Scope
In Scope
Out of Scope
Screenshots/Recordings:
Not applicable (backend enhancement)
Additional Notes:
This feature is optional and backward compatible. Deterministic mode prioritizes reproducibility over diversity and is intended for debugging, testing, and consistent evaluation scenarios.
AI Usage Disclosure:
We encourage contributors to use AI tools responsibly when creating Pull Requests. While AI can be a valuable aid, it is essential to ensure that your contributions meet the task requirements, build successfully, include relevant tests, and pass all linters. Submissions that do not meet these standards may be closed without warning to maintain the quality and integrity of the project. Please take the time to understand the changes you are proposing and their impact. AI slop is strongly discouraged and may lead to banning and blocking. Do not spam our repos with AI slop.
Check one of the checkboxes below:
I have used the following AI models and tools: ChatGPT (for guidance and review)
Checklist
Summary by CodeRabbit
Release Notes
New Features
Bug Fixes