Query: Use `llm` program instead of `cratedb-about ask`

[amotl]

About

We started with a custom implementation cratedb-about ask, but it would be way better to use a standard one. I think Simon Willison's llm program would be a good choice.

References

• https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-2025-37899-a-remote-zeroday-vulnerability-in-the-linux-kernels-smb-implementation/ (discuss)

[amotl]

First explorations using llm. The LLM prompt can be augmented using the --fragment or --system-fragment options. However, maximum context lengths are an issue.

export OPENAI_API_KEY=sk-XJZ7ofog5GpIT--INVALID--bkFJ0CJ5lyAKSefZKdV1Y3S0
uv pip install llm

It works well with individual documentation pages.

llm --sfragment https://cratedb.com/docs/sqlalchemy-cratedb/_sources/index.rst.txt "How do I use pandas with CrateDB?"

On the full llms.txt output, the default models and others fail.

$ llm --fragment llms-full.txt "How do I use pandas with CrateDB?"
Error: Error code: 400 - {'error': {'message': "This model's maximum context length is 128000 tokens. However, your messages resulted in 212787 tokens. Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}

$ llm --model o4-mini --fragment llms-full.txt "How do I use pandas with CrateDB?"
Error: Error code: 400 - {'error': {'message': "This model's maximum context length is 200000 tokens. However, your messages resulted in 212786 tokens. Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}

It works well when explicitly selecting gpt-4.1.

$ llm --model gpt-4.1 --fragment llms-full.txt "How do I use pandas with CrateDB?"

[amotl]

It works really well. Here are more examples including an "instructions" file that should make it into the documentation.

• Prompt: Add instructions about working with CrateDB #52 (comment)

instructions="https://raw.githubusercontent.com/crate/about/c3421748df4aa65703a292fdfac7d81dc8f9df24/src/cratedb_about/instruction/cratedb-instructions.md"
llm --model gpt-4.1 --fragment $instructions "How do I work with date intervals?"

[amotl]

The llm program may also get MCP support in the near future.

• Feature request: ability to use MCP servers simonw/llm#696

[amotl]

Using models from Azure AI works by using the llm-azure plugin. ¹

Footnotes

1. Recent versions of lllm might need a patch: https://github.com/simonw/llm/issues/178#issuecomment-2951178466. ↩

[amotl]

This patch makes a start to advertise the llm package.

• Prompt: Educate about standalone use with llm program #53

That's a backlog item idea.

• Make parameters accept Python module references, let's pitch that for llm?

  uvx --with=cratedb-about llm --system-fragment="cratedb_about.prompt.default" ...

[amotl]

Anthony Shaw provides the llm-github-models package, which is a plugin for llm that uses GitHub Models via the Azure AI Inference SDK.

[amotl]

Chat to various large language models through a GUI.

• https://github.com/alikalik9/openllm