Currently, our pipeline uses a centralized knowledge base that aggregates all company meta data, files, web search data, and more. We want to introduce a new feature that allows users to select specific documents to act as context for pipeline execution, rather than defaulting to all available data.
Key requirements:
- Allow users to pick one or multiple documents to serve as context for a given pipeline run.
- The pipeline should execute using only the user-selected documents as context (e.g., in the marketing pipeline, a single chosen document can be the only context).
- Consider how this affects what is saved, including output storage and possible future document selection for reproducibility.
- Maintain flexibility so default behavior can still utilize the full knowledge base if desired.
Benefits:
- Precision in processing, reducing the likelihood of noise from unrelated documents.
- Custom workflows for teams or use-cases that require tailored context.
Request:
- Design and implement a user-friendly mechanism for document selection within the existing pipeline workflow.
- Ensure the pipeline properly respects these context boundaries and only operates on selected documents.
- Document the changes and clarify how saved outputs relate to the selected context.
Currently, our pipeline uses a centralized knowledge base that aggregates all company meta data, files, web search data, and more. We want to introduce a new feature that allows users to select specific documents to act as context for pipeline execution, rather than defaulting to all available data.
Key requirements:
Benefits:
Request: