security : limiting write access to the repository/organization

[ggerganov]

@ggml-org/core @ggml-org/maintainers @ggml-org/hf Heads up, as a security precaution to the developing supply-chain attack of the LiteLLM package (BerriAI/litellm#24512), I have disabled the write permission of all existing collaborators. Please double-check that you have not been affected by the attack.

Check script:

• security : limiting write access to the repository/organization #20954 (reply in thread)
• security : limiting write access to the repository/organization #20954 (comment)

Update

Taking some extra measures to improve the security of the project:

• Disabled some of the self-hosted runners until they are properly sandboxed

• Write access of maintainers is restored. Increased "Required approvals" for PRs from 1 to 2:

  

TODOs

• Double-check that we don't have any package dependencies (both python and node) that are not bound to a specific version (cc @CISC, @allozaur)
• Move the release workflow from the master branch to a new release branch. New releases will be triggered manually, after an audit by the maintainers
• Create guidelines for securing self-hosted runners (security : limiting write access to the repository/organization #20954 (reply in thread))

[taronaeo]

Looks like I've been affected by the permissions change. I can't trigger CIs or merge PRs as before. Can you double check my permissions?

[angt]

I don’t store any SSH key on my machine nor any token. I have an old litellm version found. I don’t even know why I have it. 😂

$ sudo find / -name "litellm*.dist-info" 2>/dev/null
/System/Volumes/Data/Users/angt/.venv/lib/python3.12/site-packages/litellm-1.75.0.dist-info

[ServeurpersoCom]

Interesting, thank you!

[ngxson]

no litellm found on my computer:

Venv: ./.local/pipx/shared
  LiteLLM not found

Venv: ./.platformio/penv
  LiteLLM not found

Venv: ./testpython/.venv
  LiteLLM not found

Venv: ./.espressif/python_env/idf6.1_py3.12_env
  LiteLLM not found

Venv: ./.venvs/csm
  LiteLLM not found

Venv: ./work/executorch_test/.venv
  LiteLLM not found

Venv: ./.venv
  LiteLLM not found

[netrunnereve]

I'm clean with no litellm on my computers, thanks for the script. I think many people still aren't aware of this.

[pcuenca]

@ngxson check_litellm.sh

@pwilkin This does not seem to detect uv environments in my system, I used a regular find like @angt suggested.

[ServeurpersoCom]

Same here, no more write

I only use Python in disposable/isolated/rootless containers. These environments have no access to GitHub or any other service. and no LiteLLM here (I use llama.cpp :)

[pwilkin]

Same, relogged off and on to GH account but no change.

[JohannesGaessler]

I have never manually installed or directly used LiteLLM. When I do local language model inference I typically do it via the llama.cpp stack. The last time I installed any LLM-related Python packages was in the context of vLLM about a month ago (which to my understanding does not come with LiteLLM preinstalled and I would think my usage predates when this attack was added). I am not seeing an installation of LiteLLM in any of my relevant Python environments.

[JohannesGaessler]

So apparently LiteLLM v1.82.7 and v1.82.8 are affected. v1.82.6 was released on March 22. 2026 so the malicious package would have had to be installed very recently, which I definitely did not do.

[ngxson]

Confirm that I no longer have write permission. I ran pip list on both my workbench and my macbook but doesn't find litellm, so I think I'm safe

[JohannesGaessler]

So it seems that LiteLLM got infected via their CI. Do we have our CI set up correctly in the sense that any tokens we use have the minimum possible scope so that even if they somehow get stolen this would not compromise anything else?

[ngxson]

In general yes, we use per-workflow permissions which generates GITHUB_TOKEN that expires as soon as the workflow is finished. The token has limited permissions.

I think we should also do frequent audit to see if our CI is up to the recommendations, especially to avoid injection attacks; for now, maybe we can ask an AI agent to do that for us? @ggerganov

[IMbackK]

I have never knowingly installed litellm, i dont have very many venvs either, none of which include any affected version.

./check_litellm.sh 
Scanning for Python virtual environments under: /home/user

Venv: ./.cache/pre-commit/repoy8slvaqj/py_env-python3.12
  LiteLLM not found

Venv: ./.cache/pre-commit/repohplr80e9/py_env-python3.12
  LiteLLM not found

Venv: ./.cache/pre-commit/repo_d4y5zeb/py_env-python3.12
  LiteLLM not found

Venv: ./.cache/pre-commit/repo9l9qz5td/py_env-python3.12
  LiteLLM not found

Venv: ./.cache/pre-commit/repoba3wa_48/py_env-python3.12
  LiteLLM not found

Venv: ./.cache/vpython-root.1000/store/python_venv-7vi80oethc6bdakqgqv889ole8/contents
  LiteLLM not found

Venv: ./.cache/vpython-root.1000/store/python_venv-114tril44ne5rukm48tdcko15s/contents
  LiteLLM not found

Venv: ./.cache/pypoetry/virtualenvs/openhands-ai-3vaqi41e-py3.12
  LiteLLM version: 1.77.7 OK

Venv: ./Programming/kiss/yolov5/venv
  LiteLLM not found

Venv: ./Programming/kiss/gpeisreg/venv
  LiteLLM not found

Venv: ./Programming/kiss/XGBoost/venv
  LiteLLM not found

Venv: ./Programming/SDImagePrepross/PersonDatasetAssembler/venv
  LiteLLM not found

Venv: ./Programming/SDImagePrepross/LLavaTagger/venv
  LiteLLM not found

Venv: ./Programming/ESP/ESP8266_RTOS_SDK/venv
  LiteLLM not found

Venv: ./Programming/beautiful-bergdoktor/venv
  LiteLLM not found

Venv: ./Documents/Devices/SonyAlphaA7/fwtool.py/venv
  LiteLLM not found

Venv: ./Documents/Devices/SonyAlphaA7/Sony-PMCA-RE/venv
  LiteLLM not found

Venv: ./aur/rocblas/src/build/virtualenv
  LiteLLM not found

Venv: ./aur/hipblaslt/src/hipBLASLt/build/virtualenv
  LiteLLM not found

Venv: ./aur/hipblaslt/src/build/virtualenv
  LiteLLM not found

Venv: ./machine-learning/tts/xtts-api-server-mantella/venv
  LiteLLM not found

Venv: ./machine-lerning/tts/TTS-WebUI/venv
  LiteLLM not found

Venv: ./machine-learning/Transformersplayground/venv
  LiteLLM not found

Venv: ./machine-learning/Diffuserspayground/venv
  LiteLLM not found

Venv: ./machine-learning/llmbot/venv
  LiteLLM not found

Venv: ./machine-learning/repos/venv
  LiteLLM not found

Venv: ./machine-learning/repos/examples/distributed/FSDP/venv
  LiteLLM not found

Venv: ./machine-lerning/repos/flash-attention/venv
  LiteLLM not found

Venv: ./machine-learning/ComfyUI/venv
  LiteLLM not found

Venv: ./machine-learning/llama.cpp/venv
  LiteLLM not found

Venv: ./machine-learning/Hunyuan3D-2/venv
  LiteLLM not found

Venv: ./machine-lerning/stt/whisper_dictation/venv
  LiteLLM not found

Venv: ./machine-learning/rag/marker/venv
  LiteLLM not found

Venv: ./machine-learning/rag/wikipedia-mcp/venv
  LiteLLM not found

Venv: ./machine-learning/Wan2GP/venv
  LiteLLM not found

Venv: ./machine-learning/mistral-vibe/venv
  LiteLLM not found

Venv: ./apps/Windows Programs/Skyrim/Mantella/venv
  LiteLLM not found

[max-krasnyansky]

All clean over here. Never used litellm but double-checked just in case.

[CISC]

No trace of LiteLLM, though I leave nothing to harvest anyway. :)

[0cc4m]

No LiteLLM to be found, anywhere.

[0cc4m]

@ggerganov I don't have any permissions currently. Are we waiting for something or is it unintentional?

[ggerganov]

I am trying to figure out what to do with the self-hosted runners. Atm, it appears they are a security risk: #20954 (reply in thread)

[ngxson]

Btw @ggml-org/security I'm just wondering, but is it worth changing the CI settings to "require approval for external contributors" ?

While existing contributors (people who had at least one merged PR) cannot trigger write action via CI run from a PR (i.e. cannot create a new release), they can, in theory, extract secrets if we have any (the last time I check, there was no secret set on CI)

There can potentially be some attack vectors to self-hosted runner, but I'm not knowledgeable enough about GH self-hosted runner implementation.

[ngxson]

Or are we thinking of persistent attacks where malware is installed on the self-hosted runners, creating another attack vector?

I think this is the bigger concern. My 2 macs and 2 sparks, although disposable without any sensitive information on them, are not well sandboxed. So as a first step I'll disable them until I fix this. @taronaeo Not sure about your runner, but feel free to disable it until it is fully sandboxed.

Same concern about this point. Although GH self-hosted runner does use docker to isolate workflow, I'm still a bit skeptical if it's well-isolated, given they explicitly tell not to use SH runner on public repo (I assume they refer to attacks where malicious code than "escape" container, like for. ex kernel driver bugs)

I'm now thinking if we should get rid of GH self-hosted runners altogether and make our own solution instead. A while ago we had ggml-org/ci which I think was quite a good direction, the idea can be:

• We run a job from hosted runner, the job trigger self-hosted machine to run a pre-defined script
• The pre-defined script is outside of the main repo, so that malicious users cannot change it
• The pre-defined script always pull master branch of the main repo, here we assume master branch is safe
• Since everything to be pulled is publicly accessible, the machine doesn't need any credentials

[pcuenca]

cc @glegendre01 in case he thinks it's useful to chime in (he has vast experience on securing CI envs)

[netrunnereve]

If we're talking about running it in Docker containers like how GitHub-hosted Runners are doing, it's possible and done before by other repositories, but it requires @/ggerganov's Personal Access Token (PAT) to constantly refresh the runner token upon every startup which induces a security risk by itself.

Hmm would something like this work?

1. Create a container and set it up as a regular self hosted runner
2. Snapshot the container
3. Have it run the job
4. When the job is finished shut down the runner and load the old snapshot to reset it

This would allow us to not have to regenerate tokens every time as Github still thinks we're still using the same machine as before, it just got restarted. I don't think they require us to get a new token on every reboot.

On another thought, I think the main concern with self-hosted runners right now is the ability to leak tokens / secrets from GitHub as they get passed through to the self-hosted runners as demonstrated with the LiteLLM attack right? Or are we thinking of persistent attacks where malware is installed on the self-hosted runners, creating another attack vector?

Do we even need to pass secrets to the self hosted runners at all? The only thing they really need is the access key to set up the runner itself and run the job, we can turn off ccache and everything. Stuff like releases, the current repository bots, and so on which actually need secrets to work can be set to only run on the Github machines.

[taronaeo]

I've been playing around with different configurations for a few days now and I think I can give some feedback now.

Create a container and set it up as a regular self hosted runner
Snapshot the container
Have it run the job
When the job is finished shut down the runner and load the old snapshot to reset it

In theory this will work because only the registration token lasts 1 hour. After the registration is completed, the connection keys are stored persistently (in .credentials_rsaparams file) and runners offline for a long time can still reconnect.

However, to shutdown the container automatically after the job is completed isn't that easy. If we're looking at the --ephemeral flag, it will automatically de-register the runner once the job is completed and will not reconnect again unless we reconfigure it with a new registration token.

If we are looking to use the env var ACTIONS_RUNNER_HOOK_JOB_COMPLETED="shutdown -h now" (supported officially by GitHub Actions), most containers do not have any way to shutdown the container from inside except to use exit 1. But if exit 1 is not ran from the parent process (i.e., ./run.sh), the container will not shutdown.

On another note, I've been looking into Actions Runner Controller (ARC) which seems to be the proper way to deploy a use-case like ours but this would require every self-hosted runner owner to create their own GitHub App (to avoid using PATs). This approach is apparently being used by the RISC-V self-hosted runner maintainers but I've not gotten it to work yet, with the runner registration somehow failing.

Let me know if you know of another approach that we can try. I'll still be researching this.

[netrunnereve]

If we are looking to use the env var ACTIONS_RUNNER_HOOK_JOB_COMPLETED="shutdown -h now" (supported officially by GitHub Actions), most containers do not have any way to shutdown the container from inside except to use exit 1. But if exit 1 is not ran from the parent process (i.e., ./run.sh), the container will not shutdown.

This is the method I was thinking of as we want to reuse the token. I know it's possible to shut down a lxc container or a vm from within with the shutdown command, I don't know about Docker as I haven't tried doing that before. Basically what would happen is that there would be a script running on the server that would regularly check for shut down containers, reload the snapshot, and then restart them so that they can pick up new jobs.

On another note, I've been looking into Actions Runner Controller (ARC) which seems to be the proper way to deploy a use-case like ours but this would require every self-hosted runner owner to create their own GitHub App (to avoid using PATs). This approach is apparently being used by the RISC-V self-hosted runner maintainers but I've not gotten it to work yet, with the runner registration somehow failing.

Yeah it looks like Github is making this really painful for us, probably in hopes that people will pay for their runners instead 😢. I don't want to make it too difficult for the people donating us machines as well.

[aldehir]

Vibecoded PowerShell script for Windows users (probably only me):
https://gist.github.com/aldehir/faad916112d97aa54ca30ea4688d823e

No installed versions of LiteLLM in any of my virtualenvs.

[LostRuins]

No litellm for me

[lhez]

No litellm found for me.

[reeselevine]

No litellm on my system!

[ggerganov]

The write access is now restored. I've updated the OP of this thread with some action items.

[EAddario]

I don't think I've any write access but for what is worth, no litellm here neither