[FEATURE] Upgrade tokenizers in crewAI to match that in latest transformer version

[jd-coderepos]

Feature Area

Integration with external tools

Is your feature request related to a an existing bug? Please link it here.

No.

Describe the solution you'd like

If I install crewAI after installing transformers, I get the following

transformers 4.51.3 requires tokenizers<0.22,>=0.21, but you have tokenizers 0.20.3 which is incompatible

crewAI seems to install an earlier tokenizers 0.20.3. Are there any plans to upgrade to the most recent transformer/tokenizer versions? Thanks!

Describe alternatives you've considered

No response

Additional context

No response

Willingness to Contribute

No, I'm just suggesting the idea

[Vidit-Ostwal]

There is a cross module dependency on embedchain,If it is possible can you downgrade your transformer module to 4.46.3

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.