You deployed a fix for Log4Shell in December 2021. Your scanners showed clean. Three months later, an attacker used the same vulnerability to compromise a legacy microservice you forgot was still running.
This isn't hypothetical. Consider a team that discovered they were running vulnerable components in production for weeks after patches existed, simply because their vulnerability detection ended at the build pipeline. The gap between "we scanned the image" and "we know what's actually running" created an exploitable window.
What Happened
A production service remained vulnerable to a publicly disclosed CVE for 18 days after the fix was available. The security team had scanned container images during build, found nothing, and moved on. The vulnerable component existed in a sidecar container deployed through a separate process that wasn't part of the standard CI/CD pipeline.
The team discovered the exposure only after seeing anomalous traffic patterns in their SIEM. By then, the attacker had established persistence.
Timeline
Day 0: CVE published for a widely-used open source library
Day 1: Security team runs scans against container registry
Day 2: Development teams begin updating dependencies in active repositories
Day 7: Main application images rebuilt and redeployed with patched versions
Day 18: SOC analyst notices unusual outbound connections from production namespace
Day 18 + 4 hours: Incident response confirms exploitation of the Day 0 CVE
Day 18 + 6 hours: Team discovers vulnerable sidecar container not covered by standard scanning
Day 18 + 18 hours: Service rebuilt, redeployed, and validated clean
Which Controls Failed
No runtime inventory: The team couldn't answer "what components are actually running in production right now?" They knew what they'd built recently, but had no persistent record of deployed artifacts.
Point-in-time scanning only: Vulnerability detection happened at build time. New CVEs published daily across open source ecosystems, but nothing connected those disclosures to running systems.
No SBOM generation or storage: Container images shipped without machine-readable component lists. When the CVE dropped, the team had to manually trace dependencies across dozens of repositories.
Incomplete deployment visibility: The sidecar deployment process bypassed standard pipelines. No one realized it was exempt from scanning until the incident.
What Standards Require
PCI DSS v4.0.1 Requirement 6.3.2 mandates that you maintain an inventory of bespoke and custom software, and third-party software components. The requirement explicitly calls out the need to know what you're running, not just what you built.
NIST Cybersecurity Framework v2.0 maps this to ID.AM-2: "Software platforms and applications within the organization are inventoried." The guidance notes that inventories must be current and include version information.
ISO/IEC 27001:2022 Control 8.9 requires you to maintain an inventory of information and other associated assets. For software systems, this means knowing your dependency tree at runtime, not just at compile time.
OWASP ASVS v4.0.3 Section 14.2 covers build and deploy requirements. V14.2.3 specifically addresses component analysis: "Verify that the build pipeline warns of out-of-date or insecure components and takes appropriate actions."
None of these standards accept "we scanned it once" as sufficient. They require ongoing visibility into what's deployed.
Lessons and Action Items
Generate SBOMs at Build Time
Start producing Software Bill of Materials documents for every deployable artifact. SBOMs serve as persistent inventories of components running in deployed systems. Tools like Syft, CycloneDX, or SPDX generators integrate into most build pipelines.
Action: Add SBOM generation as a mandatory build step. Store the output alongside your container images in your registry. Tag it with the same version identifier.
Store SBOMs in a Queryable System
An SBOM sitting in an S3 bucket helps no one during an incident. You need to answer "which production services use libfoo version 1.2.3?" in under 60 seconds.
Action: Deploy an SBOM repository that supports queries by component name, version, and CVE identifier. Dependency-Track and Bomber are open source options that handle this.
Map Deployments to SBOMs
Your Kubernetes cluster knows what images are running. Your SBOM repository knows what's in each image. Connect them.
Action: Write a controller that watches your cluster's deployed images and correlates them with stored SBOMs. When a new CVE publishes, query this mapping to find affected workloads immediately.
Implement Continuous Vulnerability Matching
New CVEs publish daily. Your SBOM database should automatically check deployed components against fresh vulnerability data.
Action: Configure your SBOM repository to pull from the National Vulnerability Database (NVD) or GitHub Advisory Database hourly. Set up alerts when a new CVE matches a component in production. National Vulnerability Database
Cover All Deployment Paths
The sidecar that caused this incident deployed through Helm charts maintained by a platform team. The security team didn't know it existed.
Action: Audit every mechanism that can deploy code to production: CI/CD pipelines, GitOps tools, Helm releases, operator-managed resources. Ensure each path generates and registers an SBOM.
Test Your Response Time
Pick a random component from your production inventory. Simulate a critical CVE disclosure. Time how long it takes to identify all affected services and deploy fixes.
Action: Run this drill quarterly. Measure: time to identify affected services, time to build patches, time to deploy, time to validate. Track these metrics over time.
Document Ownership
When you discover a vulnerable component, you need to know who can fix it. The SBOM tells you what's broken. Your CMDB should tell you who owns it.
Action: Link SBOM component data to service ownership records. During an incident, you should be able to generate a list of affected services with owner contact information in one query.
The gap between "we scanned the code" and "we know what's running" is where incidents happen. SBOMs don't prevent vulnerabilities, but they let you find and fix them before attackers do. Start generating them today.
