Two Malicious Versions: How a Stored API Token Compromised PyPI's Lightning Package

Incident Overview

Versions 2.6.2 and 2.6.3 of the lightning package on PyPI were compromised with malicious code. An attacker exploited a stored API token to publish these versions without authorization. The malicious code acted as a credential stealer, using the Bun JavaScript runtime to extract sensitive data from systems that installed the compromised packages. PyPI has since quarantined both versions. Snyk issued advisory SNYK-PYTHON-LIGHTNING-16323121, rating the compromise with a CVSS score of 9.3 (Critical).

Timeline of Events

1. An attacker accessed a stored API token with publishing permissions for the lightning package.
2. The attacker published version 2.6.2 containing the malicious payload.
3. Version 2.6.3 was also published with the same compromise.
4. The compromise was detected and reported to PyPI.
5. PyPI quarantined both malicious versions.
6. Snyk published their security advisory.

The absence of a manual approval process allowed these versions to be published immediately.

Failed or Missing Controls

Long-lived API token without rotation: The attack exploited a stored API token that had been valid long enough for an attacker to discover and use it. There was no evidence of expiration or rotation policies.

Missing manual approval gate: The compromised versions were published directly to PyPI without human review.

Insufficient access monitoring: The unauthorized use of the API token wasn't detected in real-time.

No multi-factor authentication requirement: The API token alone was sufficient to publish.

Lack of integrity verification: The build and publish process didn't validate that the package contents matched expected artifacts from the official build pipeline.

Relevant Standards

NIST 800-53 Rev 5 addresses these failures:

• IA-5(1): Requires rotation policies and complexity requirements for token generation.
• AC-2(1): Mandates automated account management, including disabling accounts after inactivity.
• SI-4: Requires system monitoring to detect attacks and indicators of potential attacks.

ISO/IEC 27001:2022 Annex A controls:

• A.9.4.3: Applies to API credentials, requiring adherence to organizational practices.
• A.9.2.1: Requires formal processes for granting access rights.

SOC 2 Type II Common Criteria:

• CC6.1: Requires logical and physical access controls.
• CC7.2: Mandates monitoring of the system to identify anomalies.

PCI DSS (if applicable):

• Requirement 8.3.2: Requires strong authentication for all users.
• Requirement 8.6.3: Mandates that authentication factors be changed at least every 90 days.

Action Items for Your Team

Implement Token Rotation

Set maximum lifetimes for all API tokens used in CI/CD. For PyPI and similar repositories, rotate tokens every 90 days or more frequently. Consider using GitHub Actions with OIDC tokens that last minutes.

Add Manual Approval Gates

Ensure that your CI pipeline builds and tests automatically, but requires human approval for publishing to production repositories. This helps catch compromised automation.

Monitor Publishing Activity

Set alerts for:

• Publishes from new IP addresses or locations.
• Publishes outside normal working hours.
• Multiple rapid publishes.
• Significant package size changes.
• New dependencies added.

Use PyPI's event stream and webhooks for monitoring.

Switch to Short-lived Credentials

Replace stored API tokens with OIDC federation where possible. Use GitHub Actions to authenticate to PyPI with short-lived tokens.

Verify Build Provenance

Sign your packages and verify signatures before publishing. Use frameworks like SLSA to ensure build integrity.

Audit Your Current Token Inventory

List every API token your team uses and assess:

• Creation and rotation dates.
• Access permissions.
• Potential for replacement with short-lived credentials.

Revoke any token older than 90 days without justification.

Test Your Detection

Simulate an unauthorized publish in a test environment to evaluate your monitoring effectiveness. Improve detection capabilities based on the results.

The lightning compromise highlights the need for robust API token management and secure CI/CD practices. Address these gaps to prevent similar attacks.