On March 24, malicious versions of LiteLLM were uploaded to PyPI for just 40 minutes. In that brief period, credential-stealing code infiltrated CI/CD pipelines at over 2,500 organizations, according to CloudSEK's analysis. This attack was part of a larger campaign linked to the Trivy scanner compromise.
The FBI has advised rotating CI/CD secrets, publishing tokens, and cloud credentials. However, simply rotating credentials isn't enough if your architecture still relies on long-lived credentials stored in configuration files for extended periods.
Timeline
March 24, 2025 (exact times not disclosed):
- Malicious LiteLLM packages published to PyPI
- Packages contained credential-exfiltration code
- Organizations with unpinned dependencies automatically pulled compromised versions
40 minutes later:
- Malicious packages removed from PyPI
- Damage already done: credentials from 2,500+ organizations potentially compromised
Post-incident:
- CloudSEK published dataset mapping organizational exposure
- FBI issued rotation guidance
- Investigation linked attack to broader Trivy scanner campaign
Which Controls Failed
Dependency pinning:
Organizations running pip install litellm without version constraints pulled whatever PyPI served. No human reviewed the code that entered production environments.
Credential lifetime management:
The stolen credentials were long-lived tokens that remained valid after exfiltration. In many cases, these credentials had been sitting in .env files or CI/CD variables for months.
Supply chain verification: Teams didn't verify package signatures or checksums before installation. The malicious code executed with the same permissions as legitimate LiteLLM code.
Transitive dependency monitoring: LiteLLM itself may have been a direct dependency, but many organizations pulled it transitively through other packages. Those indirect relationships weren't tracked or reviewed.
What Standards Require
NIST 800-53 Rev 5, IA-5(7) requires that authenticators have "limited lifetimes" and are "changed or refreshed" at defined intervals. Long-lived API tokens that persist for months violate this control.
PCI DSS v4.0.1, Requirement 8.3.2 mandates that authentication credentials are "changed at least once every 90 days." Many of the exposed credentials in this incident were likely older than that threshold.
ISO/IEC 27001:2022, Control 5.17 (Authentication information) requires that "allocation and management of authentication information shall be controlled, including advice to users on appropriate handling of authentication information." Storing long-lived tokens in environment files doesn't meet this bar.
SOC 2 Type II, CC6.1 (Logical and Physical Access Controls) requires that "the entity implements logical access security measures to protect against threats from sources outside its system boundaries." Credentials that remain valid indefinitely expand the attack surface beyond what's necessary.
Lessons and Action Items
Replace long-lived credentials with temporary ones:
AWS temporary security credentials expire in 15 minutes to 12 hours. GCP service account tokens can be scoped to one hour. Azure managed identities rotate automatically. If you're storing credentials in .env files, you're doing it wrong.
Start with your CI/CD pipelines. GitHub Actions supports OIDC tokens that authenticate to cloud providers without storing secrets. GitLab CI can do the same. The credential never touches your configuration.
Pin your dependencies, including transitive ones:
Run pip freeze > requirements.txt after you've verified your environment works. Don't let PyPI decide what code runs in production.
For transitive dependencies, use tools like pip-tools to generate a fully resolved dependency tree. Review it. A new transitive dependency is a new attack surface.
Monitor credential usage: If a credential gets exfiltrated, you need to know. Set up alerts for API tokens used from unexpected IP ranges or geographic locations. AWS CloudTrail, GCP Cloud Audit Logs, and Azure Monitor can all flag anomalous access patterns.
For GitHub tokens, enable push protection and secret scanning. It won't catch everything, but it'll catch tokens committed to repositories.
Implement break-glass procedures: You need a way to rotate credentials across your entire infrastructure in under an hour. The malicious LiteLLM packages were live for 40 minutes. If your rotation process takes longer than that, you're already behind.
Document which credentials live where. Map them to the services that depend on them. Test your rotation procedure quarterly. When you need it, you won't have time to figure it out.
Verify packages before installation: PyPI supports package signing, but adoption is low. Until it's universal, verify checksums against known-good values. Store those checksums in version control, separate from your dependency declarations.
For critical dependencies, consider vendoring the code and reviewing it manually. It's labor-intensive, but it's the only way to guarantee you know what's running.
Audit your current credential inventory:
Right now, list every long-lived credential in your infrastructure. API keys in .env files, service account tokens in Kubernetes secrets, database passwords in configuration management systems.
For each one, ask: Does this need to be long-lived? Can we replace it with a temporary credential? Can we scope it more narrowly?
Then build a migration plan. You don't have to fix everything today, but you need to know what needs fixing.
The LiteLLM incident proved that 40 minutes is enough time to compromise thousands of organizations. Long-lived credentials give attackers months or years to use what they've stolen. That's not a security model; it's a countdown timer.
