Preliminary Report: Oct 21st Security Incident | Railway Blog
Railway's detailed postmortem of a 38-hour breach by an AI-powered attacker hunting for model distillation keys reveals the new reality of cyber threats: adversaries with abliterated models and massive compute targeting AI infrastructure specifically.
Read Original Summary used for search
TLDR
• Attacker with "abliterated models and substantial compute resources" executed sophisticated exploits specifically to steal AI provider API keys for model distillation
• 1.85% of workspaces (~100k) potentially affected through privilege escalation in build infrastructure and eventual decryption of envelope-encrypted secrets via compromised CI credentials
• Railway immediately shut down builds, notified vendors (GitHub, OpenAI, Anthropic, etc.) within hours, and accelerated rollout of 4th-gen single-tenant builder infrastructure
• Attack timeline: Oct 20 13:58 UTC (first detection) to Oct 22 03:32 UTC (confirmed attacker ejection), with full transparency including what they don't yet know
• Represents new threat class: AI-powered attackers targeting AI infrastructure, requiring industry-wide elevation of security postures beyond traditional approaches
In Detail
Railway experienced a targeted 38-hour security incident where an attacker with access to abliterated AI models and substantial compute resources executed sophisticated exploits to steal AI provider API keys for model distillation attacks. The attacker gained initial access through privilege escalation in Railway's build infrastructure, eventually compromising CI environment credentials that allowed decryption of envelope-encrypted environment variables. Forensic evidence shows the attacker specifically targeted workspaces with large numbers of environment variables and high concentrations of AI vendor keys, representing a new class of threat where AI capabilities are weaponized against AI infrastructure.
Railway's response demonstrates modern incident handling at scale: immediate infrastructure shutdown prioritizing user safety over availability, proactive notification to vendors (GitHub, GitLab, OpenAI, Anthropic, OpenRouter, SpaceXSI, Stripe) within hours of confirmed access, and engagement of external forensics experts. They accelerated deployment of their planned 4th-generation single-tenant builder infrastructure (originally scheduled for the following week) and performed complete credential rotation across all systems. The company provided granular transparency including exact UTC timestamps, specific attack vectors, and clear delineation between confirmed facts and ongoing investigation areas.
The incident affected approximately 1.85% of workspaces (~100,000), though Railway proactively shared all potentially affected keys with vendors out of abundance of caution, resulting in broader revocations than actual confirmed compromises. Critically, running user deployments remained online throughout—only new builds were paused during remediation. Railway frames this as a watershed moment requiring the industry to recognize that security is no longer one team's concern but "everyone's ground floor" as AI agents become integral to software development. Their response included not just remediation but fundamental infrastructure upgrades and acknowledgment as "critical cyber infrastructure" enabling access to previously unavailable security tooling.