feat: setup benchmarking

[isSerge]

• I have looked for existing issues (including closed) about this

Feature Request

Setup bencmarking and cover CPU- and Memory-bound tasks like vector search, schema generation, document parsing, etc. (specific list of tasks can be defined separately).

Motivation

• Validate Rig's position as a high-performance alternative to other AI Frameworks (mostly Python- and JavaScript-based)
• Tracking benchmarks will help identify bottlenecks and prevent any performance regressions

Proposal

• Use criterion.rs or divan crate to setup benchmarking
• CI/CD integration: use CodSpeed or Bencher to run benchmarks in CI and generate a repor, attach it to PRs
• Cover any single CPU-heavy task, for example in-memory vector search or JSON schema generation
• More tasks can be covered separately based on priorities

Alternatives

[linear]

RIG-1279

[gold-silver-copper]

I generally like this idea, benchmarks are always fun. It would probably require a decent amount of setup and careful test planning though. To test against both Javascript and Python libraries I assume they would have to setup the proper JS and Python environments. So we would have to choose which do we use: node? deno? bun? Python also has performance differences between say CPython and PyPy.

CI is probably not the best place for this, we use free github CI runners which have a limited amount of minutes per month, and are on fairly underpowered, sometimes throttled CPUs. Local benchmarks might be better, so that users can see what's better on their own machine, because performance can sometimes vary drastically per machine. Ideally this would all be self contained in some sort of container, maybe docker, nix, etc

Sometimes it is very hard to write fair tests, because libraries could be doing many different things under the hood, for various reasons. Another thing to consider is what server will they be testing against, almost certainly not a live provider. So maybe a dummy server that sends fake responses?

What are your thoughts on this?

[isSerge]

1. I was thinking about Rig only benchmarks. Maybe other frameworks already have published benchmarks (I did not check tbh), either way it is nice to have. Teams and enterprises currently using LangChain or similar frameworks and face performance and scale limitations might switch to Rig if benchmarks are provided.
2. Live provider won't be useful. It should be isolated and use dummy data

[gold-silver-copper]

Having rig only benchmarks could be useful for regression testing but I don't think that is a high priority right now, we don't want to optimize prematurely.

For teams currently using LangChain or other frameworks, I think they would not be impressed by Rig-only benchmarks. They would need to see Rig beating LangChain in very comparable, realistic workloads.

So while I think the idea is good and something we definitely want for Rig in the future, I think benchmarks are something we should hold off on for now.

[isSerge]

Fair point, just want to clarify - my point was that setting up benchmarks for Rig is the first step. Once it's done we can compare with other frameworks and do other stuff, depending on priorities

[gold-silver-copper]

Makes sense to me I just don't want to start on this without having a really good understanding of how it should all look like. I'm going to keep this open so others can share their thoughts here too.