Skip to main content
SolarWinds: What Failed and How to Prevent ItIncident
5 min readFor Security Engineers

SolarWinds: What Failed and How to Prevent It

In December 2020, security teams discovered that attackers had compromised SolarWinds' Orion platform and distributed malicious updates to roughly 18,000 customers. The breach went undetected for months. While we don't have complete public details about every control failure, we can map what we know to standard requirements and extract lessons you can apply to your own supply chain risk program.

What Happened

Attackers inserted malicious code into SolarWinds' Orion software updates. When customers installed these updates through normal patch management processes, they granted attackers access to their networks. The compromised updates bypassed traditional defenses because they came through a trusted vendor channel.

The attack highlights a critical gap: supply chain attacks enter through the front door. Your firewall doesn't question a signed update from your monitoring vendor.

Timeline

We don't have precise public dates for every phase, but the general sequence matters:

  • Compromise phase: Attackers gained access to SolarWinds' build environment.
  • Distribution phase: Malicious updates shipped to customers over several months.
  • Detection: FireEye discovered the breach after detecting unusual activity in their own environment.
  • Disclosure: Public disclosure came months after the initial compromise.

The extended dwell time is the critical failure. Organizations ran compromised software for months before anyone detected the problem.

Which Controls Failed or Were Missing

Vendor Security Assessment

Most affected organizations likely had SolarWinds on their approved vendor list, but continuous monitoring of that vendor's security posture was absent. You can't assess a vendor once during procurement and assume they remain secure.

Build Environment Isolation

The attackers reached SolarWinds' build systems. This suggests insufficient segmentation between development, build, and production environments. If you can't verify the integrity of your vendor's build pipeline, you can't trust what they ship.

Code Signing Validation

The malicious updates were properly signed, which meant signature verification alone provided no protection. Organizations trusted the signature without additional validation of the update's contents or behavior.

Anomaly Detection

Traditional intrusion detection failed because the malicious traffic looked like legitimate Orion communications. Organizations lacked behavioral monitoring that could flag unusual patterns even from trusted applications.

Immutable Backups

When organizations discovered the compromise, recovery speed depended on having clean, immutable backups that predated the infection. Many organizations couldn't quickly restore to a known-good state because their backups were either compromised or not properly isolated.

What the Standards Require

NIST Cybersecurity Framework v2.0

GV.SC-03: Cyber supply chain risk management plans include provisions for activities that occur after the conclusion of a partnership or service agreement.

You need documented procedures for monitoring vendor security throughout the relationship, not just at onboarding. This means:

  • Quarterly security questionnaires
  • Annual third-party audits
  • Real-time monitoring of vendor security incidents

ID.SC-04: Suppliers and third-party partners of information systems, components, and services are identified, prioritized, and assessed using a cyber supply chain risk assessment process.

Map every vendor that touches your production environment. Assign risk tiers based on access level and data exposure. SolarWinds would score high because it had network visibility and privileged access.

ISO/IEC 27001:2022

Control 5.19: Information security in supplier relationships requires organizations to define and agree upon security requirements with each supplier based on the type of relationship.

Your vendor contracts should specify:

  • Required security controls (MFA, encryption, build pipeline security)
  • Incident notification timelines (24 hours for potential compromise)
  • Audit rights (annual penetration test results)
  • Termination clauses triggered by security failures

Control 8.24: Use of privileged utility programs shall be restricted and tightly controlled.

Your monitoring tools have privileged access. You must implement compensating controls:

  • Network segmentation that limits lateral movement
  • Application allowlisting that restricts what monitoring agents can execute
  • Behavioral monitoring that flags unusual activity even from trusted processes

NIST 800-53 Rev 5

SR-3: Supply Chain Controls and Processes requires organizations to employ security and privacy controls to manage supply chain risks.

Document your supply chain security program:

  1. Inventory all third-party software with production access
  2. Risk-tier each vendor (critical/high/medium/low)
  3. Define monitoring requirements per tier
  4. Test recovery procedures quarterly

CP-9: Information System Backup mandates that organizations conduct backups of system-level information and test backup information to verify media reliability.

Your backup strategy must account for supply chain compromise:

  • Store backups offline or in immutable storage
  • Maintain multiple restore points spanning 90+ days
  • Test restoration monthly, not annually
  • Document recovery time objectives for each critical system

Lessons and Action Items for Your Team

Map Your Critical Vendors

Create a spreadsheet listing every vendor with:

  • Network access level
  • Data access scope
  • Update mechanism (automatic, manual, signed)
  • Last security assessment date
  • Contract termination notice period

Prioritize vendors that push automatic updates or have broad network visibility. These are your SolarWinds-equivalent risks.

Implement Continuous Vendor Monitoring

Don't wait for annual reviews. Set up:

  • RSS feeds for vendor security advisories
  • Automated checks for vendor SOC 2 Type II expiration dates
  • Monthly review of vendor security incidents reported in the news

When a vendor has a breach, you need to know within hours, not months.

Segment Monitoring Tools

Your monitoring agents shouldn't have unrestricted network access. Configure:

  • Network ACLs limiting outbound destinations to vendor-specified IPs
  • Application allowlisting preventing arbitrary code execution
  • Separate VLANs for management traffic

If your monitoring tool gets compromised, segmentation limits the blast radius.

Test Your Backup Recovery

Schedule a quarterly drill:

  1. Simulate a vendor compromise scenario
  2. Identify affected systems
  3. Restore from immutable backups
  4. Measure time to recovery
  5. Document gaps

You discover backup failures during drills, not during actual incidents.

Require Build Pipeline Attestation

For critical vendors, contract language should require:

  • SLSA framework compliance (at minimum Level 2)
  • Signed build provenance
  • Third-party audit of build environments annually

You can't prevent every vendor compromise, but you can raise the bar.

Monitor Trusted Applications

Deploy behavioral monitoring that flags:

  • Unusual outbound connections from monitoring agents
  • Privilege escalation attempts
  • Lateral movement patterns
  • Off-hours activity from service accounts

Traditional signature-based detection won't catch supply chain attacks. You need anomaly detection.

The SolarWinds breach succeeded because organizations treated vendor software as implicitly trustworthy. Your supply chain security program must assume vendors can be compromised and build defense-in-depth accordingly. Map your dependencies, monitor continuously, segment aggressively, and test your recovery procedures before you need them.

Topics:Incident

You Might Also Like