Scanning Every Package Won't Save You From Supply Chain Attacks

Organizations often rely on automated scanning tools to detect malicious packages in their dependency chains. The belief is that if you can scan quickly and block threats within seconds, you're protected. This idea gained traction after Sonatype's tools detected and blocked compromised litellm versions shortly after their PyPI publication. It seemed like automated defenses caught the threat before it could spread. Problem solved, right?

Not quite.

The Real Issue: Trust, Not Just Detection

The litellm incident reveals a harsh truth: by the time your scanning tools detect a malicious package, it might be too late. Those compromised versions were available on PyPI for at least two hours. If your team ran pip install litellm during that time—perhaps in a CI/CD pipeline or on a developer's machine—you could have downloaded a credential stealer before any scanner could intervene.

Supply chain security is not just about detection; it's about trust. When you install a package, you're granting it execution privileges in your environment. It gains access to your filesystem, environment variables, and API keys. Once that package runs, it can exfiltrate credentials to domains like models[.]litellm[.]cloud or checkmarx[.]zone. Scanners can't prevent that initial execution; they can only minimize the damage by catching subsequent installations.

This is especially critical for AI infrastructure packages. The litellm library connects your application code with multiple LLM providers and handles API keys for services like OpenAI and Anthropic. A compromised version doesn't just steal one credential—it can compromise your entire AI operations.

The Evidence from the Litellm Case

In the litellm case, the attacker published malicious versions to PyPI that acted as both credential stealers and droppers. These versions maintained normal functionality while running malicious code in the background. Sonatype's detection was impressive but reactive. Their tools identified the threat seconds after publication, but that's after the malicious code was already available. Any organization with automated dependency updates, or any developer running a quick pip install, risked infection during that window.

The attack targeted a package with over 11,000 stars on GitHub and millions of downloads. This wasn't an obscure library; it was critical infrastructure code.

Practical Steps for Robust Supply Chain Security

1. Dependency Pinning: Your requirements.txt or pyproject.toml should specify exact versions, not ranges:

   litellm==1.48.7
   

   This prevents automatic upgrades to malicious versions. You control when dependencies change.

2. Hash Verification: Use PIP's support for hash verification in requirements files:

   litellm==1.48.7 --hash=sha256:abc123...
   

   This ensures installation fails if the package contents don't match the expected hash, preventing attackers from delivering malicious code.

3. Isolate Installation from Execution: Use multi-stage Docker builds to separate package installation from execution:

   # Build stage - install dependencies
   FROM python:3.11 AS builder
   RUN pip install --user litellm==1.48.7
   
   # Runtime stage - copy only what's needed
   FROM python:3.11-slim
   COPY --from=builder /root/.local /root/.local
   

   This limits access during installation.

4. Monitor Dependency Updates: If using tools like Dependabot or Renovate, ensure human review before merging. Check:

   • When was this version published?
   • What changed in the release notes?
   • Does the diff include unexpected files or obfuscated code?

   For critical packages, consider a 72-hour waiting period before adopting new versions.

5. Runtime Monitoring: Implement monitoring for your AI infrastructure. Watch for:

   • Unexpected network connections
   • Unusual environment variable access
   • File system modifications in read-only directories

The Role of Automated Scanning Tools

Automated scanning tools are valuable for:

• Discovering known vulnerabilities in existing dependencies. If a CVE is announced for a package you installed months ago, your scanner will catch it.
• Blocking widespread attacks after initial detection. Once Sonatype identified the malicious litellm versions, their tools prevented thousands of subsequent installations.
• Meeting compliance requirements. SOC 2 auditors expect evidence of dependency scanning. NIST CSF includes supply chain risk management in its Govern function. These tools help demonstrate due diligence.

However, scanning alone is not enough. Think of it as your smoke detector. It alerts you to fires but doesn't prevent them. You need fire-resistant materials, sprinkler systems, and evacuation plans. Similarly, your AI supply chain needs a defense-in-depth approach. Pin your dependencies, verify hashes, isolate installation from execution, and monitor runtime behavior. Use automated scanning as one layer in a comprehensive strategy.

The litellm incident won't be the last supply chain attack on AI infrastructure. Packages that aggregate access to multiple LLM providers are high-value targets. Your security posture should reflect this reality—not the comforting myth that fast scanning solves everything.