Lessons Learned

Abstract risks become real when you see what happens when things go wrong. This section collects case studies of supply chain failures—what happened, why it mattered, and what the industry learned.

The Classics¶

Early incidents that shaped how we think about dependency management.

The Vulnerabilities¶

Critical security incidents that demonstrated the stakes.

Build Pipeline Failures¶

When the threat isn't an attacker — it's your own release process.

Ongoing Threats¶

Attack patterns that continue to evolve.

The Big Picture¶

Synthesis pieces connecting dots across individual incidents.

Common Threads¶

Reading these cases, patterns emerge:

Trust is transitive. When you trust a dependency, you're trusting everyone who can push code to it—maintainers, CI systems, package registries. You're also trusting their dependencies, and their dependencies' dependencies. When Trivy was compromised, every CI pipeline running trivy-action became an attack vector—Cisco didn't trust Trivy's attacker, but their build system did.

Incentives matter. Maintainer burnout leads to abandoned projects or hostile takeovers. Lack of corporate support for open source isn't just unfair—it's a security risk.

Your security tools are attack surface. Scanners, linters, and CI actions run with elevated privileges by design—they need access to your code, your secrets, your build environment. That makes them the highest-value targets in your pipeline. The Trivy breach demonstrated this perfectly: compromise the scanner, and you compromise everyone who trusts it.

Build defaults aren't deployment defaults. Development tools optimize for developer experience, not production safety. Bun generates source maps by default—great for debugging, catastrophic when it ships 512,000 lines of proprietary source to npm. Every build tool's defaults deserve scrutiny before they touch a release pipeline.

Visibility saves you. Organizations that could answer "are we affected by log4shell?" quickly had SBOMs and vulnerability scanning. Those that couldn't were scrambling for weeks.

Attacks are sophisticated. The xz backdoor took years of social engineering. SolarWinds compromised a build system. The Trivy attackers chained together a GitHub Actions footgun, a stolen PAT, incomplete credential rotation, and tag poisoning across multiple repositories before pivoting to downstream targets. These aren't script kiddies—they're patient, funded, and methodical.

Not every incident has an attacker. The most likely thing to cost you a week is an unpinned base image, a rarely-exercised release path, and a green dashboard measuring the wrong thing. No adversary, no CVE, nobody negligent — just drift, arriving on a schedule you don't control. Pin the build environment with the same discipline you pin dependencies.

A signature is not a review. Provenance attestations, sigstore signatures, and verified badges answer one question: was this artifact produced by the pipeline it claims? They say nothing about whether the source that pipeline consumed was clean. The keyv worm shipped with valid OIDC and SLSA provenance because the attacker compromised the repository and let the honest builder do the signing — then generated fresh, verifiable attestations for every package it republished. Verify signatures. Just don't mistake verification for safety.

Pin to immutable references. Mutable version tags in GitHub Actions, floating version ranges in package managers, latest tags in container images—all of these let an attacker substitute malicious code without changing anything you explicitly chose. Pin to commit SHAs. Pin to exact versions. Use npm ci in CI, not npm install. If it can change without your knowledge, it will.

State actors use your supply chain to reach their actual targets. UNC1069 didn't want Axios. They wanted the cryptocurrency and finance developers whose CI runners hold AWS credentials and private key material. The package with 70 million weekly downloads is the delivery mechanism. Your team's CI pipeline is the target.

Defense is possible. Lock files, SBOMs, vulnerability scanning, update policies, code review—none of these are perfect, but together they reduce risk substantially. Add CI checks that verify what actually ships. Auto-merge security patches so you're not choosing between velocity and safety. Use the same AI-assisted tools attackers use, but for defense.

Each case study includes sources. Where possible, we link to primary sources (CVE records, original disclosure posts, official advisories) rather than news coverage.