What Happened
Malicious versions of LiteLLM, a Python middleware for LLM API calls, were available on PyPI for about two hours. These versions contained malware designed to steal cloud provider keys and CI/CD secrets. With LiteLLM present in 36% of cloud environments, the risk was significant.
Attackers published the malicious versions to PyPI, the Python Package Index. Any automated dependency update, CI/CD pipeline run, or manual installation during that time downloaded code that immediately started harvesting credentials.
This incident is part of the broader TeamPCP supply-chain campaign, which previously targeted Trivy, a vulnerability scanner. The pattern shows a focus on security and infrastructure tools that have elevated privileges and access to sensitive credentials.
Timeline
Hour 0: Malicious LiteLLM versions published to PyPI
Hour 0-2: Packages available for download via pip install, automated dependency updates, and CI/CD pipeline executions
Hour 2: Malicious versions removed from PyPI
Post-removal: Investigation reveals base64-encoded Python code designed to exfiltrate credentials
The two-hour window is both encouraging and sobering. The quick takedown limited exposure, but two hours is enough for thousands of automated pipeline runs across organizations using LiteLLM.
Which Controls Failed or Were Missing
Package Verification
Most organizations pulling from PyPI lack runtime verification of package integrity. Your CI/CD pipeline likely runs pip install -r requirements.txt without validating:
- Package signatures
- Known-good hashes
- Publication timestamps or version age
- Sudden version bumps outside normal release cadence
Credential Exposure in Runtime Environments
The malware succeeded because credentials were available in the execution environment. Common patterns that enable this include:
- Cloud provider credentials in environment variables (AWS_ACCESS_KEY_ID, AZURE_CLIENT_SECRET)
- CI/CD secrets passed as environment variables to build steps
- Service account keys mounted or stored in accessible locations
- Kubeconfig files with cluster admin access
Behavioral Monitoring
The malicious code used base64-encoded Python for obfuscation—a technique that should trigger behavioral analysis. Missing controls include:
- No monitoring of outbound connections from build environments
- No detection of base64 decode operations followed by exec() calls
- No alerting on unexpected network destinations from CI/CD runners
Dependency Pinning and Lock Files
Organizations using unpinned dependencies (litellm>=1.0.0) or running automated dependency updates without approval gates pulled the malicious version immediately.
What the Standards Require
PCI DSS v4.0.1 Requirement 6.3.2
"Security of bespoke and custom software and software components developed for the organization is managed throughout a software development lifecycle."
This includes your dependency management process. If you're pulling arbitrary versions from public repositories into systems that process cardholder data, you're not managing security throughout the lifecycle. You need:
- Defined sources for software components
- Verification mechanisms before use
- Change control for dependency updates
NIST 800-53 Rev 5 SA-12: Supply Chain Protection
"The organization protects against supply chain threats by employing security safeguards as part of a comprehensive, defense-in-depth information security strategy."
Specific controls that apply:
- SA-12(1): Acquisition strategies that include security requirements for developers
- SA-12(5): Limitation of harm from supply chain threats
- SA-12(9): Operations security controls in external systems
For PyPI dependencies, this translates to treating public package repositories as untrusted external systems requiring verification.
ISO/IEC 27001:2022 Control 8.30: Outsourced Development
"The organization shall direct, monitor and review the activities related to outsourced system development."
Open-source dependencies are outsourced development. You're running code you didn't write. Control 8.30 requires you to:
- Define security requirements for externally developed code
- Monitor for security issues
- Review before deployment
Lessons and Action Items for Your Team
Implement Dependency Pinning with Hash Verification
Stop using version ranges in production. Your requirements.txt should look like:
litellm==1.47.0 \
--hash=sha256:abc123...
Generate this with pip-compile and pip-tools. Update dependencies through a controlled process, not automatically.
Isolate Build Environments from Credentials
Your CI/CD runners should not have direct access to production credentials. Implement:
- Workload identity federation: Let your CI/CD platform authenticate to cloud providers without long-lived keys
- Just-in-time credential provisioning: Request credentials only when needed, with minimal scope
- Separate build and deploy phases: Build artifacts in an isolated environment, then deploy from a separate runner with credentials
Monitor Build Environment Behavior
Implement detection for:
- Outbound connections to unexpected destinations during builds
- Base64 decode followed by exec() or eval()
- Environment variable enumeration
- File access outside the project directory
Tools like Falco or custom eBPF probes can provide this visibility.
Establish a Dependency Review Process
Before updating any dependency:
- Check the package publication date—versions published in the last 24 hours warrant extra scrutiny
- Review the changelog for unexpected changes
- Compare the package hash against multiple sources
- Run in an isolated environment first
For critical dependencies like authentication libraries or cloud SDKs, add a 48-hour waiting period for new versions.
Create an Incident Response Plan for Supply Chain Compromises
When a dependency is compromised, you need to answer within hours:
- Which systems pulled the malicious version?
- What credentials were available in those environments?
- What outbound connections occurred during execution?
Build this capability now. Tag your container images with their dependency versions. Log all pip install operations with timestamps. Maintain an inventory of which credentials exist in which build environments.
The LiteLLM incident shows that two hours is enough time for significant exposure. Your response needs to be faster.
