The Script That Ate Itself (2026)¶

The Lesson: ssh reads standard input. If your deploy script is arriving on standard input, ssh will eat the rest of it, and your shell will die parsing a sentence that no longer has an ending. Nothing was compromised, nothing was misconfigured, and the pipeline had worked for months.

What Happened¶

The guide you're reading deploys through a CI pipeline. Push to main and it builds and deploys internally; cut a tag and it publishes to the public site. In May, that pipeline worked. In August, a tag produced this:

+ rsync -az docker-compose.production.yml deploy@"$DROPLET_HOST":.../_staged/
+ rsync -az caddy/site.caddy deploy@"$DROPLET_HOST":.../_staged/
/bin/sh: syntax error: unterminated quoted string

Every command ran. Both file transfers succeeded. Then the shell announced it had found an unterminated quote — in a script that had no unterminated quotes, that had not been edited, and that had deployed successfully three months earlier.

Meanwhile, three months of finished, tested content sat unpublished, and every pipeline run on main reported green. It was: those runs only exercised the internal deploy. The tag-gated production path was the broken one, and nothing ran it, so nothing reported it.

The Mechanism¶

Here is the whole thing, and it is beautiful in the way a good bug is beautiful.

The CI runner does not hand a step's script to the shell as a file. It pipes it in on standard input — morally, echo "$CI_SCRIPT" | base64 -d | /bin/sh -e.

The shell reads that pipe in chunks. BusyBox ash — the /bin/sh in an Alpine container — pulls roughly 2KB per read. Our generated script was 2,222 bytes. Just over the line. So the shell read the first chunk, began executing, and left the tail sitting in the pipe to be read later.

Partway through that first chunk was this command:

ssh -o StrictHostKeyChecking=accept-new deploy@"$HOST" "mkdir -p /opt/projects/site/_staged"

ssh reads standard input and forwards it to the remote command. That's not a bug; it's the entire reason cat file | ssh host 'cat > file' works. But this ssh inherited its stdin from the shell — and the shell's stdin was the pipe carrying the rest of the script.

So ssh drained it. It read the remainder of the deploy script out of the pipe and shipped it to a remote mkdir, which ignored it.

The shell finished its buffer, reached for the next chunk, and got EOF — mid-way through the final quoted line. It had an opening quote and no closing one, because the closing one had been eaten and sent to another machine.

/bin/sh: syntax error: unterminated quoted string

The script fed itself to ssh and then starved.

# Reads the FIRST line, then exits — ssh ate the rest of the input
while read host; do
    ssh "$host" uptime
done < hosts.txt

Why May Worked and August Didn't¶

Nothing in the repository changed. git diff across the two tags touched neither the pipeline config nor the deploy script — byte for byte identical.

The container did. The step ran on this:

image: alpine:latest    # ← the actual defect

That image was re-pulled two days after the last successful deploy, and the new one shipped OpenSSH 10.2 in place of 9.x — a version that drains inherited stdin more eagerly than its predecessor. From that pull forward, every release was already dead. Nobody noticed for three months because nobody cut a tag for three months.

The bug wasn't introduced by a commit. It was introduced by a docker pull nobody ran on purpose, in a pipeline that was only exercised quarterly. There is no diff to review, no PR to blame, and no test that would have caught it — the script was correct the entire time.

latest is not a version. It's a subscription to someone else's changes, billed at a time of their choosing. This guide has been saying that about container images and mutable tags in a security context for years. It is exactly as true when nobody is attacking you.

Two Wrong Diagnoses¶

The instructive part isn't the bug. It's how confidently it got misdiagnosed, twice.

Wrong diagnosis #1 — blame the command shape. When this first appeared in May, the response was to rewrite the offending step into a different form and cut a new release. That "fixed" it in the sense that the pipeline went green — but the earlier failure had contained the same stdin-eating ssh, so the rewrite had addressed a shape, not a mechanism. It came back.

Wrong diagnosis #2 — mine. I looked at the same evidence and concluded that the unpinned Alpine image had drifted and that BusyBox ash had changed its parsing behavior.

That was right about the layer and wrong about the mechanism, which is the most dangerous kind of wrong. The image drift was real. The three-month gap was real. "Nothing in the repo changed, so the environment did" was sound reasoning that pointed at the right container for the right reason — and then named the wrong program inside it. A tidy story that explains the evidence is not the same thing as the cause, and the tidier it is, the less it invites checking.

Worse, I proposed a fix: transfer the script to the remote host and execute it there, eliminating the stdin redirect that I believed was implicated.

That fix would have worked, and it would have been a disaster. It removes the victim — the script that was being eaten — while leaving the eater armed. The pipeline would have gone green, the incident would have closed, and the next generated script that crossed 2KB would have failed identically, months later, with all the evidence of the previous investigation now pointing at a resolved ticket.

A passing pipeline is not a working one. It only means nothing failed in the way you were watching for.

The actual fix came from someone who refused to reason from the evidence and instead reproduced it: rebuilding the generated script at the same byte offsets with a deliberate stdin-consumer substituted for ssh, producing the identical error at the identical position. That's the difference between a hypothesis that explains the failure and a demonstration that causes it.

The Fix¶

image: alpine:3.23          # pinned — no more silent re-pulls

ssh -n -o StrictHostKeyChecking=accept-new deploy@"$HOST" "mkdir -p ..."
#   ^^ stdin from /dev/null; cannot consume the script

Plus a comment in the file explaining the trap, so that the next person to encounter a "fragile-looking" command shape doesn't revert the fix for the third time. That comment is load-bearing. Two previous engineers looked at that line and saw something to clean up.

Why It Mattered¶

Pin your build environment, not just your dependencies. Every team that carefully pins application dependencies will happily run CI steps on alpine:latest, ubuntu:latest, or node:latest. The build environment is a dependency. It's the one with root, your secrets, and your signing keys, and it's usually the only one nobody pinned.

Rarely-run pipelines rot silently. A path that executes quarterly has a three-month detection window by construction. If a code path only runs on release, it is effectively untested between releases — and it will break during the drift, not during the run. Exercise release paths on a schedule, or accept that you'll discover their state at the worst possible moment.

Green does not mean working. Every push to main reported success for three months while the publish path was broken, because those runs never touched it. A dashboard full of green checkmarks measured everything except the thing that mattered. Know which of your checks actually exercises the path you care about.

Plausible is not proven — especially from an AI. I produced a confident, well-structured, evidence-consistent root cause that was wrong in a way that would have produced a passing pipeline. If the response to my analysis had been "that sounds right, ship it," the bug would still be armed and the investigation would be closed. It was caught because someone reproduced the failure instead of accepting an explanation of it.

That generalizes well past this incident. The failure mode of a confident analysis is not that it's obviously wrong — it's that it's almost right, internally consistent, and pointed one layer off. Ask for the reproduction, not the reasoning.

Which is a code review problem wearing a CI costume. The habit that catches this is the one in Development Practices: review the change, not the explanation of the change.

Timeline¶

The Two-Character Version¶

- ssh -o StrictHostKeyChecking=accept-new deploy@"$HOST" "mkdir -p /opt/projects/site/_staged"
+ ssh -n -o StrictHostKeyChecking=accept-new deploy@"$HOST" "mkdir -p /opt/projects/site/_staged"