Problem

The current tokenizers~=0.20.3 constraint in crewAI prevents using recent versions of transformers (4.51+), which require tokenizers>=0.21.

transformers 4.53.3 requires tokenizers<0.22,>=0.21, but crewai requires tokenizers~=0.20.3

Analysis

After investigating the crewAI codebase:

1. crewAI does not directly import or use tokenizers - no import tokenizers or from tokenizers found in the source
2. tokenizers is listed as a direct dependency but appears to be a transitive constraint
3. chromadb 1.1.0 (which crewAI uses) only requires tokenizers>=0.13.2 with no upper bound
4. PR Fix #2782: Resolve tokenizers/transformers dependency conflict #2784 moved tokenizers to optional dependency for docling extra, but the core constraint remains

Proposed Solution

Change in lib/crewai/pyproject.toml:

- tokenizers~=0.20.3
+ tokenizers>=0.20.3

This would allow crewAI to work with both older and newer versions of transformers.

Impact

This change would enable users to use:

• transformers 4.51+ (which requires tokenizers >=0.21)
• Latest sentence-transformers
• Other ML libraries that depend on recent tokenizers

Related

• Issue [FEATURE] Upgrade tokenizers in crewAI to match that in latest transformer version #2782 - Originally reported this conflict
• PR Fix #2782: Resolve tokenizers/transformers dependency conflict #2784 - Partially addressed by making tokenizers optional for docling

Environment

• crewai: 1.7.1
• transformers: 4.53.3 (cannot install due to conflict)
• Python: 3.11