Skip to content

Too few successful evaluation #109

Open
Open
Too few successful evaluation#109
@shackmann

Description

@shackmann
shackmann
opened on Jun 6, 2025

Describe the bug
We are seeing the message "Too few successful evaluations" a bit too often and need to understand the root cause. The TrialPruned exception is raised when the number of successful evaluation is below the minimum success rate for reporting evaluation results, which is by default 50%.

To Reproduce
Here are a few configs that cause the issue:

{'additional_context_enabled': False,
 'critique_agent_llm': 'google/gemma-3-27b-it',
 'few_shot_embedding_model': 'sentence-transformers/all-MiniLM-L12-v2',
 'few_shot_enabled': True,
 'few_shot_top_k': 11,
 'hyde_enabled': True,
 'hyde_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'max_iterations': 11,
 'rag_embedding_model': 'mixedbread-ai/mxbai-embed-large-v1',
 'rag_fusion_mode': 'dist_based_score',
 'rag_method': 'dense',
 'rag_mode': 'critique_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'rag_query_decomposition_num_queries': 2,
 'rag_top_k': 4,
 'reflection_agent_llm': 'Qwen/Qwen2.5',
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.25,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'subquestion_response_synthesizer_llm': 'google/gemma-3-27b-it',
 'template_name': 'default'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 17,
 'few_shot_embedding_model': 'BAAI/bge-multilingual-gemma2',
 'few_shot_enabled': True,
 'few_shot_top_k': 3,
 'hyde_enabled': False,
 'rag_embedding_model': 'BAAI/bge-multilingual-gemma2',
 'rag_method': 'dense',
 'rag_mode': 'rag',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 7,
 'reranker_enabled': True,
 'reranker_llm_name': 'Qwen/Qwen2.5',
 'reranker_top_k': 2,
 'response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.4,
 'splitter_method': 'recursive',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'lats_max_rollouts': 4,
 'lats_num_expansions': 3,
 'rag_embedding_model': 'sentence-transformers/all-MiniLM-L12-v2',
 'rag_method': 'dense',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 5,
 'reranker_enabled': True,
 'reranker_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'reranker_top_k': 2,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.30000000000000004,
 'splitter_method': 'token',
 'template_name': 'default'}
{'additional_context_enabled': False,
 'few_shot_enabled': False,
 'hyde_enabled': True,
 'hyde_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'max_iterations': 10,
 'rag_embedding_model': 'BAAI/bge-large-en-v1.5',
 'rag_fusion_mode': 'relative_score',
 'rag_hybrid_bm25_weight': 0.7,
 'rag_method': 'hybrid',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'rag_query_decomposition_num_queries': 5,
 'rag_top_k': 3,
 'reranker_enabled': True,
 'reranker_llm_name': 'google/gemma-3-27b-it',
 'reranker_top_k': 3,
 'response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.1,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'subquestion_response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'template_name': 'default'}
{'additional_context_enabled': False,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'rag_embedding_model': 'BAAI/bge-small-en-v1.5',
 'rag_fusion_mode': 'dist_based_score',
 'rag_method': 'dense',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'rag_query_decomposition_num_queries': 5,
 'rag_top_k': 7,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'google/gemma-3-27b-it',
 'splitter_chunk_exp': 7,
 'splitter_chunk_overlap_frac': 0.4,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'template_name': 'CoT'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 2,
 'few_shot_embedding_model': 'BAAI/bge-multilingual-gemma2',
 'few_shot_enabled': True,
 'few_shot_top_k': 4,
 'hyde_enabled': False,
 'rag_fusion_mode': 'dist_based_score',
 'rag_method': 'sparse',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'rag_query_decomposition_num_queries': 3,
 'rag_top_k': 8,
 'reranker_enabled': True,
 'reranker_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'reranker_top_k': 26,
 'response_synthesizer_llm': 'Qwen/Qwen2.5',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.35000000000000003,
 'splitter_method': 'sentence',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'template_name': 'CoT'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'TencentBAC/Conan-embedding-v1',
 'few_shot_enabled': True,
 'few_shot_top_k': 3,
 'hyde_enabled': False,
 'rag_embedding_model': 'BAAI/bge-large-en-v1.5',
 'rag_method': 'dense',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 2,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.45,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'google/gemma-3-27b-it',
 'template_name': 'CoT'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 10,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'rag_embedding_model': 'sentence-transformers/all-MiniLM-L12-v2',
 'rag_method': 'dense',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 4,
 'reranker_enabled': True,
 'reranker_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'reranker_top_k': 80,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.45,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'Qwen/Qwen2.5',
 'subquestion_response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'template_name': 'CoT'}
{'additional_context_enabled': False,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'max_iterations': 11,
 'rag_embedding_model': 'mixedbread-ai/mxbai-embed-large-v1',
 'rag_fusion_mode': 'dist_based_score',
 'rag_hybrid_bm25_weight': 0.4,
 'rag_method': 'hybrid',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'rag_query_decomposition_num_queries': 2,
 'rag_top_k': 9,
 'reranker_enabled': True,
 'reranker_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'reranker_top_k': 3,
 'response_synthesizer_llm': 'google/gemma-3-27b-it',
 'splitter_chunk_exp': 7,
 'splitter_chunk_overlap_frac': 0.30000000000000004,
 'splitter_method': 'token',
 'subquestion_engine_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'subquestion_response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'sentence-transformers/all-MiniLM-L12-v2',
 'few_shot_enabled': True,
 'few_shot_top_k': 15,
 'hyde_enabled': True,
 'hyde_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'max_iterations': 11,
 'rag_embedding_model': 'BAAI/bge-base-en-v1.5',
 'rag_method': 'dense',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 1,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'google/gemma-3-27b-it',
 'splitter_chunk_exp': 10,
 'splitter_chunk_overlap_frac': 0.30000000000000004,
 'splitter_method': 'token',
 'subquestion_engine_llm': 'google/gemma-3-27b-it',
 'subquestion_response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'template_name': 'CoT'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'BAAI/bge-large-en-v1.5',
 'few_shot_enabled': True,
 'few_shot_top_k': 14,
 'hyde_enabled': False,
 'rag_embedding_model': 'Linq-AI-Research/Linq-Embed-Mistral',
 'rag_fusion_mode': 'simple',
 'rag_method': 'dense',
 'rag_mode': 'rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'google/gemma-3-27b-it',
 'rag_query_decomposition_num_queries': 2,
 'rag_top_k': 9,
 'reranker_enabled': True,
 'reranker_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'reranker_top_k': 25,
 'response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'splitter_chunk_exp': 7,
 'splitter_chunk_overlap_frac': 0.15000000000000002,
 'splitter_method': 'recursive',
 'template_name': 'concise'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 13,
 'few_shot_embedding_model': 'thenlper/gte-large',
 'few_shot_enabled': True,
 'few_shot_top_k': 5,
 'hyde_enabled': True,
 'hyde_llm_name': 'Qwen/Qwen2.5',
 'rag_fusion_mode': 'simple',
 'rag_method': 'sparse',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'rag_query_decomposition_num_queries': 5,
 'rag_top_k': 1,
 'reranker_enabled': True,
 'reranker_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'reranker_top_k': 15,
 'response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.05,
 'splitter_method': 'token',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'google/gemma-3-27b-it',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'sentence-transformers/all-MiniLM-L12-v2',
 'few_shot_enabled': True,
 'few_shot_top_k': 11,
 'hyde_enabled': False,
 'rag_embedding_model': 'thenlper/gte-large',
 'rag_fusion_mode': 'simple',
 'rag_hybrid_bm25_weight': 0.2,
 'rag_method': 'hybrid',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 1,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'google/gemma-3-27b-it',
 'splitter_chunk_exp': 7,
 'splitter_chunk_overlap_frac': 0.5,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'Qwen/Qwen2.5',
 'template_name': 'default'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 2,
 'few_shot_embedding_model': 'Linq-AI-Research/Linq-Embed-Mistral',
 'few_shot_enabled': True,
 'few_shot_top_k': 14,
 'hyde_enabled': True,
 'hyde_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'max_iterations': 11,
 'rag_embedding_model': 'mixedbread-ai/mxbai-embed-large-v1',
 'rag_fusion_mode': 'reciprocal_rerank',
 'rag_method': 'dense',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'google/gemma-3-27b-it',
 'rag_query_decomposition_num_queries': 5,
 'rag_top_k': 6,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.2,
 'splitter_method': 'token',
 'subquestion_engine_llm': 'Qwen/Qwen2.5',
 'subquestion_response_synthesizer_llm': 'google/gemma-3-27b-it',
 'template_name': 'CoT'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'Snowflake/snowflake-arctic-embed-l-v2.0',
 'few_shot_enabled': True,
 'few_shot_top_k': 5,
 'hyde_enabled': True,
 'hyde_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'max_iterations': 10,
 'rag_embedding_model': 'BAAI/bge-base-en-v1.5',
 'rag_fusion_mode': 'simple',
 'rag_method': 'dense',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'rag_query_decomposition_num_queries': 2,
 'rag_top_k': 1,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'google/gemma-3-27b-it',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.2,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'template_name': 'CoT'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 10,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'rag_embedding_model': 'BAAI/bge-multilingual-gemma2',
 'rag_fusion_mode': 'relative_score',
 'rag_method': 'dense',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'rag_query_decomposition_num_queries': 5,
 'rag_top_k': 7,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.45,
 'splitter_method': 'token',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'Qwen/Qwen2.5',
 'template_name': 'CoT'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 11,
 'few_shot_embedding_model': 'BAAI/bge-small-en-v1.5',
 'few_shot_enabled': True,
 'few_shot_top_k': 11,
 'hyde_enabled': False,
 'rag_embedding_model': 'BAAI/bge-small-en-v1.5',
 'rag_method': 'dense',
 'rag_mode': 'rag',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 10,
 'reranker_enabled': True,
 'reranker_llm_name': 'google/gemma-3-27b-it',
 'reranker_top_k': 14,
 'response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.1,
 'splitter_method': 'recursive',
 'template_name': 'CoT'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 3,
 'few_shot_embedding_model': 'BAAI/bge-large-en-v1.5',
 'few_shot_enabled': True,
 'few_shot_top_k': 19,
 'hyde_enabled': False,
 'max_iterations': 10,
 'rag_embedding_model': 'Linq-AI-Research/Linq-Embed-Mistral',
 'rag_method': 'dense',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 10,
 'reranker_enabled': True,
 'reranker_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'reranker_top_k': 6,
 'response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.1,
 'splitter_method': 'sentence',
 'subquestion_engine_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'subquestion_response_synthesizer_llm': 'Qwen/Qwen2.5',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'sentence-transformers/all-MiniLM-L12-v2',
 'few_shot_enabled': True,
 'few_shot_top_k': 8,
 'hyde_enabled': True,
 'hyde_llm_name': 'google/gemma-3-27b-it',
 'rag_method': 'sparse',
 'rag_mode': 'rag',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 2,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.30000000000000004,
 'splitter_method': 'sentence',
 'template_name': 'CoT'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 6,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'rag_embedding_model': 'BAAI/bge-small-en-v1.5',
 'rag_fusion_mode': 'reciprocal_rerank',
 'rag_method': 'dense',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'rag_query_decomposition_num_queries': 4,
 'rag_top_k': 9,
 'reranker_enabled': True,
 'reranker_llm_name': 'google/gemma-3-27b-it',
 'reranker_top_k': 90,
 'response_synthesizer_llm': 'Qwen/Qwen2.5',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.0,
 'splitter_method': 'token',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'lats_max_rollouts': 5,
 'lats_num_expansions': 3,
 'rag_embedding_model': 'Linq-AI-Research/Linq-Embed-Mistral',
 'rag_fusion_mode': 'simple',
 'rag_method': 'dense',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'google/gemma-3-27b-it',
 'rag_query_decomposition_num_queries': 3,
 'rag_top_k': 8,
 'reranker_enabled': True,
 'reranker_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'reranker_top_k': 6,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.05,
 'splitter_method': 'token',
 'template_name': 'default'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 11,
 'few_shot_embedding_model': 'TencentBAC/Conan-embedding-v1',
 'few_shot_enabled': True,
 'few_shot_top_k': 4,
 'hyde_enabled': True,
 'hyde_llm_name': 'Qwen/Qwen2.5',
 'rag_embedding_model': 'Linq-AI-Research/Linq-Embed-Mistral',
 'rag_fusion_mode': 'dist_based_score',
 'rag_hybrid_bm25_weight': 0.2,
 'rag_method': 'hybrid',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'google/gemma-3-27b-it',
 'rag_query_decomposition_num_queries': 2,
 'rag_top_k': 2,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'google/gemma-3-27b-it',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.5,
 'splitter_method': 'sentence',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'thenlper/gte-large',
 'few_shot_enabled': True,
 'few_shot_top_k': 13,
 'hyde_enabled': False,
 'lats_max_rollouts': 2,
 'lats_num_expansions': 2,
 'rag_embedding_model': 'Snowflake/snowflake-arctic-embed-l-v2.0',
 'rag_fusion_mode': 'simple',
 'rag_hybrid_bm25_weight': 0.4,
 'rag_method': 'hybrid',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'Qwen/Qwen2.5',
 'rag_query_decomposition_num_queries': 3,
 'rag_top_k': 8,
 'reranker_enabled': True,
 'reranker_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'reranker_top_k': 14,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.15000000000000002,
 'splitter_method': 'recursive',
 'template_name': 'CoT'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'BAAI/bge-large-en-v1.5',
 'few_shot_enabled': True,
 'few_shot_top_k': 3,
 'hyde_enabled': True,
 'hyde_llm_name': 'Qwen/Qwen2.5',
 'max_iterations': 10,
 'rag_embedding_model': 'BAAI/bge-multilingual-gemma2',
 'rag_fusion_mode': 'reciprocal_rerank',
 'rag_hybrid_bm25_weight': 0.7,
 'rag_method': 'hybrid',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 1,
 'reranker_enabled': True,
 'reranker_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'reranker_top_k': 102,
 'response_synthesizer_llm': 'Qwen/Qwen2.5',
 'splitter_chunk_exp': 7,
 'splitter_chunk_overlap_frac': 0.1,
 'splitter_method': 'sentence',
 'subquestion_engine_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'subquestion_response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'template_name': 'CoT'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'BAAI/bge-multilingual-gemma2',
 'few_shot_enabled': True,
 'few_shot_top_k': 4,
 'hyde_enabled': True,
 'hyde_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'lats_max_rollouts': 5,
 'lats_num_expansions': 3,
 'rag_embedding_model': 'sentence-transformers/paraphrase-multilingual-mpnet-base-v2',
 'rag_method': 'dense',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 2,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.35000000000000003,
 'splitter_method': 'token',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'BAAI/bge-base-en-v1.5',
 'few_shot_enabled': True,
 'few_shot_top_k': 19,
 'hyde_enabled': True,
 'hyde_llm_name': 'Qwen/Qwen2.5',
 'max_iterations': 11,
 'rag_fusion_mode': 'simple',
 'rag_method': 'sparse',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'rag_query_decomposition_num_queries': 5,
 'rag_top_k': 6,
 'reranker_enabled': True,
 'reranker_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'reranker_top_k': 26,
 'response_synthesizer_llm': 'google/gemma-3-27b-it',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.1,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'Qwen/Qwen2.5',
 'subquestion_response_synthesizer_llm': 'Qwen/Qwen2.5',
 'template_name': 'default'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 10,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'rag_embedding_model': 'TencentBAC/Conan-embedding-v1',
 'rag_fusion_mode': 'dist_based_score',
 'rag_hybrid_bm25_weight': 0.2,
 'rag_method': 'hybrid',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'google/gemma-3-27b-it',
 'rag_query_decomposition_num_queries': 4,
 'rag_top_k': 9,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'Qwen/Qwen2.5',
 'splitter_chunk_exp': 10,
 'splitter_chunk_overlap_frac': 0.15000000000000002,
 'splitter_method': 'token',
 'subquestion_engine_llm': 'google/gemma-3-27b-it',
 'subquestion_response_synthesizer_llm': 'Qwen/Qwen2.5',
 'template_name': 'concise'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 12,
 'few_shot_enabled': False,
 'hyde_enabled': True,
 'hyde_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'lats_max_rollouts': 4,
 'lats_num_expansions': 2,
 'rag_embedding_model': 'BAAI/bge-multilingual-gemma2',
 'rag_method': 'dense',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 10,
 'reranker_enabled': True,
 'reranker_llm_name': 'Qwen/Qwen2.5',
 'reranker_top_k': 62,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 10,
 'splitter_chunk_overlap_frac': 0.4,
 'splitter_method': 'token',
 'template_name': 'default'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'BAAI/bge-small-en-v1.5',
 'few_shot_enabled': True,
 'few_shot_top_k': 8,
 'hyde_enabled': False,
 'max_iterations': 10,
 'rag_fusion_mode': 'relative_score',
 'rag_method': 'sparse',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'google/gemma-3-27b-it',
 'rag_query_decomposition_num_queries': 3,
 'rag_top_k': 8,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'Qwen/Qwen2.5',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.5,
 'splitter_method': 'token',
 'subquestion_engine_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'subquestion_response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'template_name': 'concise'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 7,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'lats_max_rollouts': 5,
 'lats_num_expansions': 2,
 'rag_fusion_mode': 'relative_score',
 'rag_method': 'sparse',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'rag_query_decomposition_num_queries': 4,
 'rag_top_k': 2,
 'reranker_enabled': True,
 'reranker_llm_name': 'google/gemma-3-27b-it',
 'reranker_top_k': 11,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.1,
 'splitter_method': 'recursive',
 'template_name': 'CoT'}
{'additional_context_enabled': False,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'rag_embedding_model': 'TencentBAC/Conan-embedding-v1',
 'rag_fusion_mode': 'simple',
 'rag_method': 'dense',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'rag_query_decomposition_num_queries': 4,
 'rag_top_k': 2,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'Qwen/Qwen2.5',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.35000000000000003,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'template_name': 'default'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 8,
 'critique_agent_llm': 'google/gemma-3-27b-it',
 'few_shot_embedding_model': 'TencentBAC/Conan-embedding-v1',
 'few_shot_enabled': True,
 'few_shot_top_k': 19,
 'hyde_enabled': False,
 'max_iterations': 10,
 'rag_embedding_model': 'sentence-transformers/all-MiniLM-L12-v2',
 'rag_fusion_mode': 'simple',
 'rag_method': 'dense',
 'rag_mode': 'critique_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'rag_query_decomposition_num_queries': 4,
 'rag_top_k': 10,
 'reflection_agent_llm': 'Qwen/Qwen2.5',
 'reranker_enabled': False,
 'response_synthesizer_llm': 'google/gemma-3-27b-it',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.5,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'google/gemma-3-27b-it',
 'subquestion_response_synthesizer_llm': 'Qwen/Qwen2.5',
 'template_name': 'concise'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 15,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'lats_max_rollouts': 4,
 'lats_num_expansions': 2,
 'rag_embedding_model': 'mixedbread-ai/mxbai-embed-large-v1',
 'rag_fusion_mode': 'reciprocal_rerank',
 'rag_hybrid_bm25_weight': 0.2,
 'rag_method': 'hybrid',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 7,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.05,
 'splitter_method': 'sentence',
 'template_name': 'CoT'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 6,
 'few_shot_enabled': False,
 'hyde_enabled': True,
 'hyde_llm_name': 'Qwen/Qwen2.5',
 'max_iterations': 10,
 'rag_embedding_model': 'sentence-transformers/all-MiniLM-L12-v2',
 'rag_fusion_mode': 'simple',
 'rag_method': 'dense',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'Qwen/Qwen2.5',
 'rag_query_decomposition_num_queries': 3,
 'rag_top_k': 2,
 'reranker_enabled': True,
 'reranker_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'reranker_top_k': 2,
 'response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.15000000000000002,
 'splitter_method': 'sentence',
 'subquestion_engine_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'subquestion_response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'critique_agent_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'max_iterations': 11,
 'rag_embedding_model': 'BAAI/bge-multilingual-gemma2',
 'rag_method': 'dense',
 'rag_mode': 'critique_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 4,
 'reflection_agent_llm': 'google/gemma-3-27b-it',
 'reranker_enabled': False,
 'response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.45,
 'splitter_method': 'sentence',
 'subquestion_engine_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'subquestion_response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'critique_agent_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'max_iterations': 11,
 'rag_method': 'sparse',
 'rag_mode': 'critique_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 1,
 'reflection_agent_llm': 'Qwen/Qwen2.5',
 'reranker_enabled': False,
 'response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.45,
 'splitter_method': 'sentence',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'template_name': 'default'}
{'additional_context_enabled': False,
 'few_shot_embedding_model': 'BAAI/bge-small-en-v1.5',
 'few_shot_enabled': True,
 'few_shot_top_k': 3,
 'hyde_enabled': False,
 'rag_fusion_mode': 'dist_based_score',
 'rag_method': 'sparse',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'rag_query_decomposition_num_queries': 3,
 'rag_top_k': 1,
 'reranker_enabled': True,
 'reranker_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'reranker_top_k': 66,
 'response_synthesizer_llm': 'Qwen/Qwen2.5',
 'splitter_chunk_exp': 10,
 'splitter_chunk_overlap_frac': 0.25,
 'splitter_method': 'sentence',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'Qwen/Qwen2.5',
 'template_name': 'concise'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 3,
 'few_shot_embedding_model': 'Snowflake/snowflake-arctic-embed-l-v2.0',
 'few_shot_enabled': True,
 'few_shot_top_k': 11,
 'hyde_enabled': True,
 'hyde_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'rag_embedding_model': 'TencentBAC/Conan-embedding-v1',
 'rag_fusion_mode': 'dist_based_score',
 'rag_method': 'dense',
 'rag_mode': 'rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'Qwen/Qwen2.5',
 'rag_query_decomposition_num_queries': 3,
 'rag_top_k': 7,
 'reranker_enabled': True,
 'reranker_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'reranker_top_k': 4,
 'response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.45,
 'splitter_method': 'token',
 'template_name': 'CoT'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 5,
 'few_shot_embedding_model': 'mixedbread-ai/mxbai-embed-large-v1',
 'few_shot_enabled': True,
 'few_shot_top_k': 15,
 'hyde_enabled': True,
 'hyde_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'lats_max_rollouts': 3,
 'lats_num_expansions': 3,
 'rag_embedding_model': 'BAAI/bge-multilingual-gemma2',
 'rag_fusion_mode': 'dist_based_score',
 'rag_hybrid_bm25_weight': 0.2,
 'rag_method': 'hybrid',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'rag_query_decomposition_num_queries': 2,
 'rag_top_k': 1,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 7,
 'splitter_chunk_overlap_frac': 0.05,
 'splitter_method': 'sentence',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'critique_agent_llm': 'microsoft/Phi-4-multimodal-instruct',
 'few_shot_enabled': False,
 'hyde_enabled': True,
 'hyde_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'max_iterations': 11,
 'rag_fusion_mode': 'relative_score',
 'rag_method': 'sparse',
 'rag_mode': 'critique_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'google/gemma-3-27b-it',
 'rag_query_decomposition_num_queries': 5,
 'rag_top_k': 2,
 'reflection_agent_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'reranker_enabled': True,
 'reranker_llm_name': 'Qwen/Qwen2.5',
 'reranker_top_k': 94,
 'response_synthesizer_llm': 'Qwen/Qwen2.5',
 'splitter_chunk_exp': 10,
 'splitter_chunk_overlap_frac': 0.35000000000000003,
 'splitter_method': 'sentence',
 'subquestion_engine_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'subquestion_response_synthesizer_llm': 'Qwen/Qwen2.5',
 'template_name': 'default'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 14,
 'few_shot_embedding_model': 'mixedbread-ai/mxbai-embed-large-v1',
 'few_shot_enabled': True,
 'few_shot_top_k': 11,
 'hyde_enabled': False,
 'rag_embedding_model': 'sentence-transformers/paraphrase-multilingual-mpnet-base-v2',
 'rag_fusion_mode': 'dist_based_score',
 'rag_hybrid_bm25_weight': 0.5,
 'rag_method': 'hybrid',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'Qwen/Qwen2.5',
 'rag_query_decomposition_num_queries': 5,
 'rag_top_k': 1,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.35000000000000003,
 'splitter_method': 'sentence',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'template_name': 'concise'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 4,
 'few_shot_embedding_model': 'sentence-transformers/paraphrase-multilingual-mpnet-base-v2',
 'few_shot_enabled': True,
 'few_shot_top_k': 10,
 'hyde_enabled': False,
 'lats_max_rollouts': 4,
 'lats_num_expansions': 3,
 'rag_embedding_model': 'Linq-AI-Research/Linq-Embed-Mistral',
 'rag_fusion_mode': 'simple',
 'rag_hybrid_bm25_weight': 0.4,
 'rag_method': 'hybrid',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 3,
 'reranker_enabled': True,
 'reranker_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'reranker_top_k': 12,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.5,
 'splitter_method': 'token',
 'template_name': 'CoT'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 5,
 'few_shot_enabled': False,
 'hyde_enabled': True,
 'hyde_llm_name': 'google/gemma-3-27b-it',
 'max_iterations': 10,
 'rag_fusion_mode': 'simple',
 'rag_method': 'sparse',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'google/gemma-3-27b-it',
 'rag_query_decomposition_num_queries': 5,
 'rag_top_k': 3,
 'reranker_enabled': True,
 'reranker_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'reranker_top_k': 3,
 'response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.2,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'subquestion_response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'template_name': 'default'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 2,
 'few_shot_embedding_model': 'sentence-transformers/paraphrase-multilingual-mpnet-base-v2',
 'few_shot_enabled': True,
 'few_shot_top_k': 8,
 'hyde_enabled': True,
 'hyde_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'rag_method': 'sparse',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 10,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'google/gemma-3-27b-it',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.15000000000000002,
 'splitter_method': 'token',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'google/gemma-3-27b-it',
 'template_name': 'default'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 5,
 'few_shot_embedding_model': 'sentence-transformers/all-MiniLM-L12-v2',
 'few_shot_enabled': True,
 'few_shot_top_k': 8,
 'hyde_enabled': True,
 'hyde_llm_name': 'google/gemma-3-27b-it',
 'max_iterations': 11,
 'rag_fusion_mode': 'simple',
 'rag_method': 'sparse',
 'rag_mode': 'react_rag_agent',
 'rag_query_decomposition_enabled': True,
 'rag_query_decomposition_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'rag_query_decomposition_num_queries': 4,
 'rag_top_k': 1,
 'reranker_enabled': True,
 'reranker_llm_name': 'nvidia/Llama-3_3-Nemotron-Super-49B',
 'reranker_top_k': 19,
 'response_synthesizer_llm': 'Qwen/Qwen2.5',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.30000000000000004,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'Qwen/Qwen2.5',
 'template_name': 'concise'}
{'additional_context_enabled': False,
 'few_shot_enabled': False,
 'hyde_enabled': True,
 'hyde_llm_name': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'lats_max_rollouts': 4,
 'lats_num_expansions': 2,
 'rag_method': 'sparse',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 4,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.5,
 'splitter_method': 'sentence',
 'template_name': 'CoT'}
{'additional_context_enabled': False,
 'few_shot_enabled': False,
 'hyde_enabled': False,
 'rag_embedding_model': 'BAAI/bge-multilingual-gemma2',
 'rag_method': 'dense',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 8,
 'reranker_enabled': True,
 'reranker_llm_name': 'microsoft/Phi-4-multimodal-instruct',
 'reranker_top_k': 66,
 'response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'splitter_chunk_exp': 7,
 'splitter_chunk_overlap_frac': 0.35000000000000003,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B',
 'template_name': 'default'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 9,
 'few_shot_embedding_model': 'BAAI/bge-large-en-v1.5',
 'few_shot_enabled': True,
 'few_shot_top_k': 8,
 'hyde_enabled': False,
 'lats_max_rollouts': 3,
 'lats_num_expansions': 3,
 'rag_embedding_model': 'Snowflake/snowflake-arctic-embed-l-v2.0',
 'rag_fusion_mode': 'simple',
 'rag_hybrid_bm25_weight': 0.5,
 'rag_method': 'hybrid',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 10,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 9,
 'splitter_chunk_overlap_frac': 0.05,
 'splitter_method': 'sentence',
 'template_name': 'CoT'}
{'additional_context_enabled': False,
 'few_shot_enabled': False,
 'hyde_enabled': True,
 'hyde_llm_name': 'google/gemma-3-27b-it',
 'rag_embedding_model': 'sentence-transformers/all-MiniLM-L12-v2',
 'rag_method': 'dense',
 'rag_mode': 'sub_question_rag',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 4,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'Qwen/Qwen2.5',
 'splitter_chunk_exp': 8,
 'splitter_chunk_overlap_frac': 0.25,
 'splitter_method': 'recursive',
 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct',
 'subquestion_response_synthesizer_llm': 'google/gemma-3-27b-it',
 'template_name': 'default'}
{'additional_context_enabled': True,
 'additional_context_num_nodes': 7,
 'few_shot_embedding_model': 'BAAI/bge-large-en-v1.5',
 'few_shot_enabled': True,
 'few_shot_top_k': 19,
 'hyde_enabled': False,
 'lats_max_rollouts': 3,
 'lats_num_expansions': 3,
 'rag_embedding_model': 'mixedbread-ai/mxbai-embed-large-v1',
 'rag_fusion_mode': 'simple',
 'rag_hybrid_bm25_weight': 0.5,
 'rag_method': 'hybrid',
 'rag_mode': 'lats_rag_agent',
 'rag_query_decomposition_enabled': False,
 'rag_top_k': 7,
 'reranker_enabled': False,
 'response_synthesizer_llm': 'microsoft/Phi-4-multimodal-instruct',
 'splitter_chunk_exp': 10,
 'splitter_chunk_overlap_frac': 0.15000000000000002,
 'splitter_method': 'token',
 'template_name': 'concise'}

In logs we see:

^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       | Traceback (most recent call last):
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/py_modules_files/_ray_pkg_9d9949a5341176cd/syftr/evaluation.py", line 181, in exception_catcher
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     results = await func(**kwargs)
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |               ^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/py_modules_files/_ray_pkg_9d9949a5341176cd/syftr/flows.py", line 149, in agenerate
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     response, duration = await self._agenerate(query, invocation_id)
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py", line 370, in async_wrapper
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     result = await func(*args, **kwargs)
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/py_modules_files/_ray_pkg_9d9949a5341176cd/syftr/flows.py", line 347, in _agenerate
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     response = await self.query_engine.aquery(query)
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py", line 370, in async_wrapper
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     result = await func(*args, **kwargs)
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/base/base_query_engine.py", line 64, in aquery
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     query_result = await self._aquery(str_or_query_bundle)
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py", line 370, in async_wrapper
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     result = await func(*args, **kwargs)
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/query_engine/sub_question_query_engine.py", line 193, in _aquery
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     qa_pairs_all = await asyncio.gather(*tasks)
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/home/alex/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     future.result()
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/home/alex/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 314, in __step_run_and_handle_result
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     result = coro.send(None)
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |              ^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |   File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/query_engine/sub_question_query_engine.py", line 227, in _aquery_subq
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |     query_engine = self._query_engines[sub_q.tool_name]
^[[36m(<lambda> pid=3670915, ip=172.16.5.148)^[[0m       |                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions