On October 20th, an alert paged a Railway engineer about API calls to an internal registry that had no business existing. Sixteen hours later the company had halted customer builds twice, rotated every credential it owned, and asked nine AI providers to revoke keys it could not yet prove had been stolen.
The final count from the incident report published on November 10th: a subset of customer variables for at most 1.85% of workspaces, roughly 100,000 of them, decrypted. Linked-account tokens for 1,852 users reset. Unauthorized spend on the stolen keys estimated at around $740,000, which Railway is now refunding or backstopping from its own balance.
And the part worth sitting with: nobody phished a finance director. Nobody dropped ransomware. The attacker came for AI provider API keys, and the market that made that worth a weekend of work is one we all built together.
The break-in, in the order it happened
The full timeline is in Railway's write-up and it reads like a horror story told in UTC timestamps.
- October 18, before anything: the attacker starts spinning up dummy accounts on the platform. Automated defences kill every attempt at this stage.
- 07:28 UTC, October 20: they exploit a previously unknown flaw in the multi-tenant build infrastructure to partially escape filesystem isolation, and read files on the host VM. That yields a credential build servers use to push images into a registry.
- 13:58 UTC: anomalous registry API calls page the on-call engineer. Root cause found within the hour. Fix merged, builds halted, attacker isolated.
- The pivot: that registry credential was meant to authorize registry pushes only. But the internal Temporal cluster used to orchestrate builds accepted it as a trusted client, so the attacker could call Temporal APIs and read build histories, including the variables passed into other people's builds. One of those snapshots was the internal monorepo. Railway's own support platform deploys from it, onto Railway itself.
- 03:26 UTC, October 21: scanning that codebase, the attacker finds an erroneously configured OIDC binding in one cloud environment and steals short-lived production cloud credentials from it.
- 04:07 to 05:21 UTC: using a longer-lived credential sitting on an orchestration machine, they connect to the production database holding envelope-encrypted environment variables and pull encrypted envelopes out. Database logs show them selecting rows whose variable names matched attacker-supplied regular expressions for AI provider keys, and prioritising customers with lots of variables.
- 04:42 to 07:15 UTC: they use the decryption credential found on that same machine to unwrap the envelopes via the key management service.
- 07:14 UTC: Railway disables all active cloud credentials, revokes static ones, kills the decryption service, and shuts down parts of the platform. Builds go dark again.
- 09:24 UTC: the attacker is fully evicted.
Three design choices that turned one bug into a heist
Strip away the drama and the failure pattern is familiar, which is exactly why it matters.
1. A narrow credential that was trusted broadly. Nothing in the build credential's design said "read other tenants' build variables." The architecture just happened to accept it as a client elsewhere. Blast radius is not defined by what you intend a key to do, it is defined by every service that accepts it.
2. Cached snapshots nobody counted as secrets. The build cache held copies of repository state. That handed the attacker a privileged map of the company and, worse, a catalogue of every route they could probe next.
3. Long-lived secrets sitting next to the data they decrypt. Envelope encryption is good design. Envelope encryption where the machine that reads the ciphertext also stores the key that unwraps it is a locked door with the key in the lock. Add one misconfigured OIDC binding and the whole chain collapses.
The attacker never needed a zero-day past the first ten minutes. Everything after was configuration.
Why they wanted your AI keys, specifically
Railway's forensics concluded that AI provider keys were the primary target. The stolen keys were passed to what the report calls distributed transfer stations, which is the polite term for resale operations that monetise someone else's inference credits. The most likely use, per the report, is distillation: generating training data and scores from a frontier model at somebody else's expense.
The report also corrects the sense that this was a one-off. OpenCode, Warp, Fireworks AI and several others disclosed similar thefts between October 6th and 23rd. Wiz documented the broader pattern as "Black October for AI developer platforms." Railway's own customers saw the stolen keys used for coding-agent requests, AI-agent behaviour reviews to build training data, and checks on training pipelines and their scoring systems.
In other words: an entire illicit supply chain now exists for inference. The keys are the currency, the spend is the loss, and the victims are often the customers whose variables lived in someone else's database.
The attacker was an agent swarm, and so was the defence
Two details in the report should change how you think about incident response timings.
First, the escalating contest on that first day was described as a contest of agent speed, with Wiz noting the intrusion used abliterated models and an agent harness similar to ARTEX. Thirty-six hours, dozens of parallel probing routes, hundreds of test accounts.
Second, Railway's countermeasure was equally machine-paced: 24/7 observation of the attacker through audit logs and traces, parallel hardening of the routes being probed, and continuous fan-out of internal forensics.
Your incident response runbook probably assumes humans on a bridge call reading Slack. That is not the pace of the other side any more.
What we would do this week
For anyone in the Gulf running AI features, agents or fintech pipelines on someone else's platform, five moves:
- Inventory every AI key and name one owner. OpenAI, Anthropic, OpenRouter, Gemini, Fireworks, the BYOK keys your agents use. If a key has no owner and no rotation date, treat it as exposed.
- Cap and alert on spend, not just failures. $740,000 of unauthorized inference is a detection opportunity. Per-key budgets, spike alerts, and a weekly review of keys that only ever spend and never deploy.
- Kill long-lived credentials. Short-lived OIDC exchanges, one audience per service, and a standing audit of every binding you have ever configured. The Railway chain started with a binding that was simply wrong.
- Assume the platform's database is readable by someone. Which of your variables contain something worth stealing? Rotate the AI keys, payment tokens and GitHub OAuth grants you stored in any platform, and prefer keys you can revoke in one click at the provider.
- Put disclosure and revocation in the contract. Railway asked providers to revoke keys using hashes before it knew what had leaked. Ask your vendors today: can you produce a hash list and a revocation path within four hours of an incident?
The line to remember
Railway should be credited for a genuinely uncomfortable, unusually detailed report. It published a preliminary note while recovery was still running and the full forensic account three weeks later, including the fact its own customer's keys cost more than a small company's annual run rate.
But the lesson is not that one platform got beaten. It is that the most valuable thing in your cloud account stopped being data and became compute. Your AI keys are money now, and they are sitting in a dozen environment variable panes across a dozen dashboards, each one encrypted with a key someone else's build server can read.
Treat them like the bank accounts they have quietly become. Rotate, scope, cap, and watch. This quarter, not next.