Skip to content

performance_tuning_tips.md

Latest commit

 

History

History
144 lines (108 loc) · 4.67 KB

performance_tuning_tips.md

File metadata and controls

144 lines (108 loc) · 4.67 KB
Raw

Performance Tuning Tips

This guide provides recommendations for optimizing the performance of your deployment.

Table of Contents

System Configuration Tips

LLM Model Selection

  • To modify the LLM model, change the llm_model in config.yaml before deploying the pipeline.
  • All supported LLM models are listed here.
# Example configuration
llm_model: "casperhansen/llama-3-8b-instruct-awq"

Vector Database Selection

  • For large-scale deployments (e.g., 1M+ vectors), use redis-cluster instead of standard redis.
  • Modify the vector_store parameter in config.yaml:
vector_databases:
  enabled: true
  namespace: vdb
  vector_store: redis-cluster  # Options: redis, redis-cluster

Component Scaling

TeiRerank Scaling

  • Match the number of TeiRerank replicas to the number of CPU sockets on your machine for optimal performance.
  • Adjust parameters in resources-reference-cpu.yaml.
# Example for a 2-socket system
teirerank:
  replicas: 2  # Set to number of CPU sockets

VLLM Scaling

  • For machines with ≤64 physical cores per socket: use 1 replica per socket
  • For machines with >64 physical cores per socket (e.g., 96 or 128): use 2 replicas per socket
  • Adjust in resources-reference-cpu.yaml.
# Example for a 2-socket system with ≤64 cores per socket
vllm:
  replicas: 2  # 1 replica per socket × 2 sockets
# Example for a 2-socket system with >64 cores per socket
vllm:
  replicas: 4  # 2 replicas per socket × 2 sockets
  • Additionally, If your machine has less then 32 physical cores per numa node, you need to reduce the number of CPU cores for vLLM:
# Example for system with only 24 cores per numa node
  vllm:
    replicas: 1
    resources:
      requests:
        cpu: 24
        memory: 64Gi
      limits:
        cpu: 24
        memory: 100Gi

Performance Tip: Consider enabling Sub-NUMA Clustering (SNC) in BIOS for better VLLM performance. This helps optimize memory access patterns across NUMA nodes.

LLM-usvc Scaling

  • When running more then one vLLM instance and when system is accessed by multiple concurrent users (e.g., 64+ users) use at least 2 replicas of llm-usvc.
  • Adjust parameters in resources-reference-cpu.yaml.
llm-usvc:
  replicas: 2

Runtime Parameter Tuning

You can adjust microservice parameters (e.g., top_k for reranker, k for retriever, max_new_tokens for LLM) using one of these methods:

  1. Using the Admin Panel UI:

    • Navigate to the Admin Panel section in the UI
    • Find detailed instructions in UI features

  2. Using Configuration Scripts:

Warning: Only parameters that don't require a microservice restart can be adjusted at runtime.

Horizontal Pod Autoscaling

  • Consider enabling HPA in order to allow the system to dynamically scale required resources in cluster.
  • HPA can be enabled in config.yaml:
hpaEnabled: true

Balloons Policy

  • Balloons Policy is responsible for assigning optimal resources for LLM inference pods such as vLLM and it is crucial for the performance of the whole deployment.
  • It can be enabled in config.yaml:
balloons:
  enabled: true

Monitoring and Validation

After making performance tuning changes, monitor system performance using:

  • The built-in metrics dashboard
  • Load testing with sample queries
  • Memory and CPU utilization metrics

This will help validate that your changes have had the desired effect.