docs: Add MLflow integration guide by debu-sinha · Pull Request #2344 · truera/trulens

debu-sinha · 2026-02-01T18:52:42Z

Summary

Adds documentation for using TruLens feedback functions as MLflow GenAI scorers.

Resolves #2343

Changes

New folder: docs/component_guides/integrations/ - For third-party integrations
New file: docs/component_guides/integrations/index.md - Integrations index
New file: docs/component_guides/integrations/mlflow.md - MLflow integration guide

Documentation includes:

Installation instructions (MLflow >= 3.10.0)
Available scorers table:
- RAG scorers: Groundedness, ContextRelevance, AnswerRelevance, Coherence
- Agent trace scorers: LogicalConsistency, ExecutionEfficiency, PlanAdherence, PlanQuality, ToolSelection, ToolCalling
Usage examples (direct calls and batch evaluation with mlflow.genai.evaluate)
Model configuration for multiple providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI)
Threshold configuration
MLflow tracing integration
Best practices and troubleshooting

Context

The TruLens integration was merged into MLflow in PR #19492 and ships in MLflow 3.10.0. This documentation helps TruLens users discover and use this integration.

Navigation

This creates a new "Integrations" section under Component Guides. The navigation may need to be updated in the site config if not auto-discovered.

Important

Adds documentation for integrating TruLens feedback functions with MLflow as GenAI scorers, including installation, usage, and troubleshooting.

Documentation:
- Adds mlflow.md in docs/component_guides/integrations/ for MLflow integration guide.
- Includes installation instructions for MLflow >= 3.10.0 and TruLens.
- Details available scorers: RAG and Agent Trace scorers.
- Provides usage examples for direct calls and batch evaluation.
- Covers model configuration for OpenAI, Anthropic, Azure, Bedrock, and Vertex AI.
- Describes threshold configuration and dynamic scorer creation.
- Explains MLflow tracing integration and viewing results.
- Lists best practices and troubleshooting tips.
Structure:
- Creates docs/component_guides/integrations/ for third-party integrations.
- Adds index.md to list available integrations.

^{This description was created by}^{for 47dc3ea. You can customize this summary. It will automatically update as commits are pushed.}

Add documentation for using TruLens feedback functions as MLflow GenAI scorers. Includes: - Installation instructions - Available scorers (RAG and Agent trace) - Usage examples (direct calls and batch evaluation) - Model configuration for multiple providers - Threshold configuration - MLflow tracing integration - Best practices and troubleshooting Resolves truera#2343 Signed-off-by: debu-sinha <debusinha2009@gmail.com>

dosubot · 2026-02-01T18:52:51Z

Related Documentation

No published documentation to review for changes on this repository.

Write your first living document

^{How did I do? Any feedback?}

debu-sinha · 2026-02-01T19:26:45Z

Note: This is a docs-only PR adding MLflow integration documentation. The sf-e2e check failure appears to be an internal Snowflake infrastructure test unrelated to documentation changes.

sfc-gh-jreini · 2026-02-02T14:55:48Z

@@ -0,0 +1,238 @@
+# MLflow Integration


Instead of creating a new /integrations/ folder, place this in docs/component_guides/evaluations.

sfc-gh-jreini

thanks again for contributing - few small things to address than can approve.

sfc-gh-jreini · 2026-02-02T14:57:07Z

+Install MLflow with TruLens support:
+
+```bash
+pip install 'mlflow>=3.10.0' trulens


will also need trulens-providers-litellm I believe

sfc-gh-jreini · 2026-02-02T14:58:19Z

@@ -0,0 +1,238 @@
+# MLflow Integration
+
+TruLens feedback functions are available as first-class scorers in MLflow's GenAI evaluation framework starting with MLflow 3.10.0. This integration was contributed by [Debu Sinha](https://github.com/debu-sinha) in [MLflow PR #19492](https://github.com/mlflow/mlflow/pull/19492).


Mentioning the PR here seems nonstandard. Can we call this out in the contribution guide instead (saying that integrating TruLens to other libraries is a new category of contributions).

sfc-gh-jreini · 2026-02-02T14:59:37Z

+| `Groundedness` | Evaluates whether the response is grounded in the provided context |
+| `ContextRelevance` | Evaluates whether the retrieved context is relevant to the query |
+| `AnswerRelevance` | Evaluates whether the response is relevant to the input query |
+| `Coherence` | Evaluates the coherence and logical flow of the response |


Coherence should be in a separate category/not limited to RAG. You could call it an Output Scorer

sfc-gh-jreini · 2026-02-02T15:04:26Z

+
+## Dynamic Scorer Creation
+
+Use `get_scorer` to create scorers dynamically:


Add a sentence or two on why you would want to create the scorers dynamically

Changes: - Move docs from integrations/ to evaluation/ folder per reviewer request - Add trulens-providers-litellm to installation instructions - Remove PR reference from intro (nonstandard) - Recategorize Coherence as "Output Scorer" (not RAG-specific) - Add explanation for dynamic scorer creation use case - Update related resources link Signed-off-by: debu-sinha <debusinha2009@gmail.com>

debu-sinha · 2026-02-02T16:04:31Z

@sfc-gh-jreini All review feedback addressed:

Moved file from integrations/ to evaluation/ folder
Added trulens-providers-litellm to installation instructions
Removed PR reference from intro
Recategorized Coherence as "Output Scorer" (separate from RAG scorers)
Added explanation for why you'd use dynamic scorer creation (get_scorer)

Ready for re-review!

sfc-gh-jreini

LGTM!

Would love to share broadly about this integration. @debu-sinha Interested in co-authoring a blog about this?

debu-sinha · 2026-02-02T16:50:46Z

Thanks for the review and glad the docs look good!

Absolutely interested in co-authoring a blog. The TruLens + MLflow integration opens up some interesting possibilities - especially the agent trace scorers covering the TRAIL evaluation framework.

Happy to contribute wherever helpful. Do you have a preferred format or platform in mind? I can draft an outline covering the key use cases (RAG evaluation, agent traces, etc.) if that would be a good starting point.

Let me know how you'd like to proceed.

sfc-gh-jreini · 2026-02-02T16:59:14Z

Thanks for the review and glad the docs look good!

Absolutely interested in co-authoring a blog. The TruLens + MLflow integration opens up some interesting possibilities - especially the agent trace scorers covering the TRAIL evaluation framework.

Happy to contribute wherever helpful. Do you have a preferred format or platform in mind? I can draft an outline covering the key use cases (RAG evaluation, agent traces, etc.) if that would be a good starting point.

Let me know how you'd like to proceed.

I've got a draft started, can I add the gmail listed on your github? Or would you prefer a different email

debu-sinha · 2026-02-02T17:51:11Z

The gmail on my GitHub works - looking forward to seeing the draft!

debu-sinha · 2026-02-10T18:36:47Z

Thanks for the blog draft and review. I finished my edits on the blog last week -- let me know if everything looks good or if anything needs adjusting.

sfc-gh-jreini · 2026-02-10T18:48:27Z

Thanks Debu, appreciate your contribution to the blog. Will reach out if anything is needed, otherwise expecting to publish this aligned with the mlflow release

dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Feb 1, 2026

dosubot Bot added the documentation Improvements or additions to documentation label Feb 1, 2026

sfc-gh-jreini reviewed Feb 2, 2026

View reviewed changes

sfc-gh-jreini requested changes Feb 2, 2026

View reviewed changes

debu-sinha requested a review from sfc-gh-jreini February 2, 2026 16:05

sfc-gh-jreini approved these changes Feb 2, 2026

View reviewed changes

sfc-gh-jreini enabled auto-merge (squash) February 2, 2026 16:19

sfc-gh-jreini disabled auto-merge February 2, 2026 17:19

sfc-gh-jreini merged commit e2c5d8c into truera:main Feb 2, 2026
2 of 3 checks passed

sfc-gh-jreini mentioned this pull request Feb 2, 2026

[MERGE ON FEB 19] Adding mlflow integration to nav #2347

Closed

6 tasks

sfc-gh-jreini mentioned this pull request Feb 2, 2026

TruLens 2.6.0 #2348

Merged

debu-sinha mentioned this pull request Feb 22, 2026

Add blog post: Agent Trace Evaluation with TruLens Scorers in MLflow mlflow/mlflow-website#482

Merged

7 tasks

		@@ -0,0 +1,238 @@
		# MLflow Integration

		TruLens feedback functions are available as first-class scorers in MLflow's GenAI evaluation framework starting with MLflow 3.10.0. This integration was contributed by [Debu Sinha](https://github.com/debu-sinha) in [MLflow PR #19492](https://github.com/mlflow/mlflow/pull/19492).


		## Dynamic Scorer Creation

		Use `get_scorer` to create scorers dynamically:

Uh oh!

Conversation

debu-sinha commented Feb 1, 2026 • edited by ellipsis-dev Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Documentation includes:

Context

Navigation

Uh oh!

dosubot Bot commented Feb 1, 2026

Uh oh!

debu-sinha commented Feb 1, 2026

Uh oh!

sfc-gh-jreini Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

sfc-gh-jreini left a comment

Choose a reason for hiding this comment

Uh oh!

sfc-gh-jreini Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

sfc-gh-jreini Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

sfc-gh-jreini Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

sfc-gh-jreini Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

debu-sinha commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sfc-gh-jreini left a comment

Choose a reason for hiding this comment

Uh oh!

debu-sinha commented Feb 2, 2026

Uh oh!

sfc-gh-jreini commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

debu-sinha commented Feb 2, 2026

Uh oh!

debu-sinha commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sfc-gh-jreini commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

debu-sinha commented Feb 1, 2026 •

edited by ellipsis-dev Bot

Loading

debu-sinha commented Feb 2, 2026 •

edited

Loading

sfc-gh-jreini commented Feb 2, 2026 •

edited

Loading

debu-sinha commented Feb 10, 2026 •

edited

Loading