Skip to main content
A Poisoned Security Scanner Backdoored a Python Package in Three HoursIncident
4 min readFor Security Engineers

A Poisoned Security Scanner Backdoored a Python Package in Three Hours

On PyPI, malicious versions of LiteLLM were available for download for about three hours. The attack vector? A compromised Trivy GitHub Action that stole the maintainer's PyPI publishing credentials. This wasn't a sophisticated zero-day exploit—it was credential theft through a tool designed to improve security.

Here's what happened, what failed, and what you need to fix in your pipeline.

What Happened

Attackers compromised the PyPI publisher credentials for LiteLLM, a Python package for interacting with LLM APIs. The compromise occurred through a poisoned version of the Trivy security scanner distributed as a GitHub Action. When the maintainer's CI/CD pipeline executed what appeared to be a legitimate security scan, the malicious action exfiltrated PyPI credentials. The attackers then published backdoored versions of LiteLLM to the official package repository.

The malicious packages were quarantined after approximately three hours of availability on PyPI. Snyk tracked the incident and issued alerts to affected users.

Timeline

  • Initial compromise: Maintainer's CI/CD pipeline runs poisoned Trivy GitHub Action.
  • Credential exfiltration: PyPI publishing token stolen during what appears to be a routine security scan.
  • Malicious upload: Attackers publish backdoored LiteLLM versions to PyPI using stolen credentials.
  • Detection window: Malicious packages remain available for approximately three hours.
  • Quarantine: PyPI removes malicious versions after detection.

Which Controls Failed or Were Missing

No credential scoping. The PyPI token had sufficient privileges to publish packages but lacked time-based or IP-based restrictions. Once stolen, it functioned identically to legitimate use.

Missing third-party verification for GitHub Actions. The pipeline consumed a GitHub Action without verifying its authenticity. GitHub Actions can be updated by their publishers at any time—your workflow references @v1 and gets whatever code the publisher decides to push to that tag.

No egress monitoring from CI/CD. The credential exfiltration went undetected. The CI/CD environment made outbound connections to attacker infrastructure without triggering alerts.

Insufficient package signing verification. While PyPI supports Trusted Publishers (which uses OIDC tokens instead of long-lived credentials), this package wasn't using that mechanism. The stolen token was sufficient to publish.

No runtime integrity checks in CI/CD. The pipeline executed arbitrary code from a third-party action without sandboxing or monitoring what that code actually did during execution.

What the Relevant Standards Require

PCI DSS v4.0.1 Requirement 6.3.2 mandates that custom scripts used in production environments are reviewed prior to deployment. GitHub Actions are executable code running in your build environment—they require the same scrutiny as any script you'd deploy.

NIST 800-53 Rev 5 SA-15 (Development Process, Standards, and Tools) requires organizations to implement security measures for tools used in the development process. A security scanner running in your CI/CD pipeline is a development tool. You need to verify its integrity and control what it can access.

ISO/IEC 27001:2022 Annex A.8.30 (Outsourced Development) addresses security requirements when using external parties in software development. GitHub Actions are external code. You need controls around their selection, verification, and monitoring.

NIST CSF v2.0 PR.DS-6 calls for integrity checking mechanisms to verify software authenticity. For GitHub Actions, this means pinning to specific commit SHAs instead of mutable tags, and verifying those commits match expected content.

SOC 2 Type II CC6.6 requires logical access controls to protect against unauthorized access. Long-lived API tokens sitting in CI/CD environments fail this control. The token should have been scoped, rotated, and ideally replaced with short-lived OIDC credentials.

Lessons and Action Items for Your Team

Pin GitHub Actions to commit SHAs, not tags. In your workflows, replace uses: aquasecurity/trivy-action@v1 with uses: aquasecurity/trivy-action@abc123def456... where the SHA is the specific commit you've reviewed. Tags can be moved. Commit SHAs cannot.

Audit your GitHub Actions dependencies now. List every action your workflows use. For each one, determine: Who maintains it? When did you last review the code? What permissions does it request? If you can't answer these questions, you have unknown code executing in your build environment with access to your secrets.

Replace long-lived tokens with Trusted Publishers. For PyPI specifically, configure Trusted Publishing using GitHub's OIDC provider. This eliminates stored credentials entirely—PyPI verifies the workflow identity directly with GitHub and issues a short-lived token. No token to steal means no token to exfiltrate.

Implement egress filtering in CI/CD. Your build runners should not be able to make arbitrary outbound connections. Create an allowlist of required destinations (your artifact repository, your container registry, specific API endpoints). Alert on anything else. This won't prevent all exfiltration, but it makes it detectable.

Separate credential scopes by purpose. If you must use long-lived tokens, create separate tokens for different packages or different types of operations. A compromised token for package A shouldn't allow publishing package B. A token for publishing shouldn't allow deleting previous versions.

Monitor package repository activity. Set up alerts for any package publication from unexpected IP addresses, at unexpected times, or with unusual version numbers. The three-hour window here was caught by external monitoring. Your internal monitoring should catch it faster.

Add SBOMs to your verification process. When you consume dependencies, verify their Software Bill of Materials if available. Changes in an SBOM between versions can indicate supply chain compromise. A security scanner that suddenly includes network libraries should raise questions.

The LiteLLM incident demonstrates that security tools themselves are attack vectors. Your pipeline trusts these tools with credentials, network access, and the ability to modify your build artifacts. Every GitHub Action is a potential backdoor. Treat it accordingly.

Software Bill of Materials (SBOM)

Topics:Incident

You Might Also Like