On 21 October at 11:39 UTC, seven wheels of litellm==1.107.0 were uploaded to PyPI. Each one carried compiled malware: a credential stealer that also fetched and loaded remote-code tools. Eight minutes later the Docker and GHCR tags latest, main-latest and main-stable pointed at the same poisoned build.
By 13:23 the package was pulled. By 14:20 the container tags were repointed.
Two hours. Long enough for a CI runner to install it, a laptop to import it, and a handful of API keys to leave the building.
If you build anything with AI, this is your stack now. Not a metaphor, not a vendor story from someone else's industry. LiteLLM is the proxy a lot of teams put in front of OpenAI, Anthropic and Bedrock to get one API, one key, one bill. When it is compromised, the compromise inherits every provider key you stored in it.
The part everyone will skip
The interesting failure is not the malware. It is how the release got signed.
LiteLLM spent the year after their March incident hardening the pipeline: releases from a private repo, Trusted GitHub Publishers, immutable tags, signed images, a publishing laptop locked in the office. By their own account, the attacker did not break any of those controls. The attacker used credentials that were already allowed to operate them:
- A GitHub CLI OAuth token belonging to a developer, which could run workflows.
- A token belonging to a GitHub organization owner, which could override branch protection and force-push a workflow change.
- A third account's credentials, used after publication to delete the workflow run records.
Where did the first token come from? From a log.
LiteLLM's investigation found that Claude Code sometimes embeds a user's GitHub CLI OAuth token inside a response body. Their proxy logged response bodies. The token ended up in the log of an internal LiteLLM deployment exposed to the public internet. An attacker with an account on that deployment read the logs, took the token, and walked into the release pipeline.
Read that chain again. Nobody phished anyone. Nobody cracked a password. A coding assistant that many teams run today put a live credential into a response it was never supposed to contain, the observability layer wrote it to disk, and the disk was reachable.
Why this is not a LiteLLM problem
The pattern generalises to every AI stack in a normal company this week:
- Access tokens move through model context. Agents paste headers, environment variables and CLI output into responses. Anything in a response can be logged, cached, traced or evaluated.
- Proxies are credential aggregators. An AI gateway holds the keys to every provider it routes to. It is, functionally, a secrets manager with an HTTP surface.
- Agents inherit developer identity. If your assistant runs with your
ghCLI login and your repo permissions, a stolen token is not "a token", it is you. - Shadow deployments have no owner. An "innocuously named user account" on an internet-facing internal deployment is the most ordinary finding in the world, and the most expensive.
The same week produced a second, independent warning. Railway disclosed that an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents, and said it believed the campaign targeted platform companies to reach AI inference providers. They asked every customer to roll every key.
Two vendors, one week, same asset class: credentials that live next to AI infrastructure.
The four controls that would have broken this chain
None of this is exotic. Each link in that chain dies to something boring.
1. Treat logs as a credential surface. Redact on write, not on read. If a response body might contain a bearer token, an OAuth token or a connection string, the logger should filter it before it reaches storage. Keep retention short for anything holding prompts and responses, and treat any log store that has ever held tokens as a rotation event.
2. Separate "publish" from "override". The release gate existed and it worked. What failed was that the same class of identity could switch it off. Branch protection an org owner can disable with one token is a suggestion with an audit trail. Publishing authority should sit behind hardware-bound, short-lived credentials and a second human, and nothing else should be able to edit the workflow that uses them.
3. Pin, then prove. pip install litellm with no version is an instruction to run whatever the internet published most recently. Pin exact versions, and pin digests for images. Then go further: a lockfile is a claim, not evidence. Generate an SBOM, verify signatures where they exist, and diff what a new version actually does in your environment.
4. Assume exposure, so the response is a checklist and not a debate. LiteLLM's own guidance is the model: if you installed it, treat the environment as compromised. Delete the dropper files, kill the daemon holding /tmp/remote_code_loader.lock, rotate the instance role, every provider key in the proxy database, the master and virtual keys, and the DATABASE_URL. Then check CloudTrail for GetCallerIdentity from the instance role at install time, and for that role's credentials appearing from outside your infrastructure.
What to do this week
If you run a LiteLLM proxy, do this before Monday:
pip show litellmon every host, image and CI runner. Anything on 1.107.0 is compromised, not "possibly compromised".- Search the filesystem for
litellm-1.107.0.dist-info,cook.pthandsitecustomize.py. Absence proves nothing; presence confirms exposure. - Rotate provider keys, master and virtual keys, the database password and the cloud instance role.
- If you run on Railway, roll your environment variables and private registry credentials.
- Audit who can edit your release workflows, and remove organization-owner tokens from anything automated.
If you would rather have someone walk the stack with you, this is the kind of work we do as a fixed engagement: credential inventory, rotation, and the pipeline controls that stop the next one. Write to [email protected].
The uncomfortable takeaway
The comfortable story is that a vendor had a bad week. The accurate story is that the modern AI stack composes three things that were never designed to sit together: a model that will happily repeat any token in its context, a proxy that stores the keys to everything, and release tooling that trusts a credential more than it trusts a person.
The malware window was two hours. The credential trail it left behind will outlive it. Fix the trail.