Uptime says green; the pipeline is broken

Here’s a failure mode that used to keep me up at night. Every unit test is green. The uptime dashboard is a wall of reassuring checkmarks. And a customer still opens a ticket because the recommendations they rely on stopped updating three days ago. Nothing “went down” — some seam between two services quietly stopped carrying data, and every alarm we had was pointed at the wrong thing. This is the story of the watchdog I built to catch that class of bug — and the decisions that kept it from becoming another flaky alert everyone learns to ignore.

The gap between “the parts work” and “it works”

I was working on a multi-tenant recommendation platform — a system that ingests each customer’s data, refreshes a model per customer, and serves ranked recommendations back to their users. It’s a pipeline: ingest → refresh the model → serve a recommendation, with a few network hops and two or three services in between.

We had the usual two layers of testing, and both were lying to us by omission.

• Unit tests proved each piece worked in isolation. The ingestion parser parsed. The serving endpoint serialized. But a unit test mocks its neighbours, so by construction it can’t see the space between the pieces — the auth token that expired, the field renamed on one side of an API but not the other.
• The health check proved each server was answering. GET /healthz returned 200, the process was up, the port was open. But “the server answers” and “the server does its job” are different claims. A recommender that returns 200 with a stale or empty payload passes every liveness probe ever written.

The bugs that reached customers lived in neither place. They lived in the integration — the full round-trip across service boundaries — which nothing exercised until a real customer did it for us. I wanted a test that behaved like a customer: feed data in one end, check a real recommendation comes out the other, continuously, in production.

The approach: a synthetic customer that never sleeps

So I gave the platform a fake customer. A dedicated synthetic tenant — a reserved account that exists only for monitoring — and a watchdog service whose whole job is to drive that tenant through the complete production pipeline on a schedule, then assert the result is sane. It’s usually called a synthetic canary: a probe that mimics a real user’s journey and keels over first when something toxic enters the mine.

The watchdog itself is almost embarrassingly simple. It’s a loop:

def run_forever(check, interval_s):
    while True:
        ok = check()                    # run one full round-trip against the synthetic tenant
        publish_status("canary", ok)    # update the health field our monitor polls
        sleep(interval_s)               # wait a few minutes, then do it all again

The interesting part is check — the round-trip. Rather than bury it in branching code, I expressed it as a straight-line pipeline of steps, each doing exactly what a real client would do at that stage:

def run_canary():
    tenant = {"id": SYNTHETIC_TENANT_ID, "name": "canary"}  # a reserved, fake account
    try:
        with canary_lock():             # only one replica runs the canary at a time
            pipe(tenant,
                 cleanup,               # delete any leftover state from a crashed prior run
                 provision,             # onboard the synthetic tenant from scratch
                 seed_known_data,       # feed in a fixed, known sample set
                 fetch_token,           # authenticate exactly like a real client
                 get_recommendation,    # call the endpoint a real user's app calls
                 assert_sane,           # is the answer shaped right, and for the right user?
                 send_feedback,         # exercise the write-back path too
                 teardown)              # offboard + delete — leave nothing behind
        return True
    except Exception as e:
        log.error(f"CANARY-FAILED: {e}")    # one failure anywhere trips the alert
        return False

If every step returns cleanly, the canary is green. If any step raises — a non-200, a failed assertion, a timeout — the whole thing is red, and the reason is captured in the log line that trips the alert. The pipeline is the test plan; you can read the sequence of steps and know exactly what “working” means.

Three decisions are what separate this from a demo.

1. Isolating the synthetic tenant so it can never touch real data

The scariest thing about a probe that writes to production is that it writes to production. A canary that seeds data and posts feedback is mechanically doing what a real ingestion job does — so the guardrail can’t be politeness, it has to be structural.

The synthetic tenant gets its own everything: a fixed, reserved tenant id that no real customer will ever be issued, and its own isolated data partition. Because the platform already routes every tenant’s data to its own storage shard, “isolation” mostly came for free — the canary’s writes land in the synthetic tenant’s shard and are physically incapable of colliding with a real customer’s rows.

The other half of isolation is time, not space. The canary cleans up before it starts, not just after it finishes. That ordering matters more than it looks:

def cleanup(tenant):
    # A previous run may have crashed halfway and left the tenant half-provisioned.
    # Deleting first means we always start from a known-empty state — the run is
    # idempotent, so a single failure can't wedge the canary permanently.
    delete_tenant_if_exists(tenant["id"])
    return tenant

If the teardown at the end of a run never happens — the process is killed, a deploy lands mid-run — the next run’s cleanup erases the debris and starts fresh. Skip cleanup-first and one crash leaves a half-provisioned tenant lying around, so every subsequent run fails trying to onboard something that already exists. It’s the difference between a canary that self-heals and one a human has to reset after every interrupted deploy.

2. Assertions that are meaningful but not flaky

This is the part that makes or breaks trust. A canary that cries wolf gets muted — and a muted canary is worse than none: a false sense of coverage and an ignored alert.

The temptation is to assert on the actual recommendation: “the top result should be item X.” Don’t. Models drift, data changes, ranking is legitimately non-deterministic — pin the output and you’ll get paged every time the model does its job slightly differently. The assertion has to be tight enough to catch a real break but loose enough to survive normal variation:

def assert_sane(ctx):
    resp = get(f"/tenants/{ctx['id']}/users/{ctx['user']}/recommendation",
               token=ctx["token"])
    if not resp.ok:
        raise CanaryError(f"recommender returned {resp.status_code}")
    body = resp.json()
    # Don't assert exact scores or a specific top item — that would be flaky.
    # Assert the answer is *for the right user* and *shaped like a recommendation*.
    if body["userId"] != ctx["user"] or not body["items"]:
        raise CanaryError("recommendation came back empty or for the wrong user")
    return ctx

Two checks, both about identity and shape rather than value. Did we get a recommendation back at all — a non-empty list? And is it addressed to the user we asked about, proving the request flowed through to the right place rather than a cache or a stray default? Feeding in a known, fixed sample set is what makes even this possible: because the input is constant, an empty result unambiguously means something broke, not this customer has no data today.

3. One lock, and an alert you can still hear

The watchdog runs with several replicas for its own availability, which creates a hazard: two replicas driving the same synthetic tenant at once would trip over each other’s cleanup and provisioning and generate phantom failures. So the round-trip runs inside a lock held in the database we already had — first replica to grab it runs the canary, the rest skip this tick. No new infrastructure, just a row saying “someone’s using the canary right now.”

The last piece is alert hygiene. A failing pipeline can stay red for an hour, and a naive alert would fire on every tick the whole time. The failure path routes through a channel that collapses repeats within a cooldown window — the first failure pages the on-call; identical ones for the next stretch are swallowed. You find out immediately, and once.

What it caught — and the payoff branch

The bugs this caught were exactly the ones the other two layers structurally couldn’t: a permission change that let the recommender answer 200 with an empty body; a serialization mismatch deployed on one service before its neighbour caught up; a misconfigured region that pointed ingestion at the wrong storage. Every one is invisible to a unit test (which mocks the neighbour) and to a health check (which only asks “are you up?”). The canary sees them because it’s the only thing making the full trip.

That’s the branch on the right of the diagram: assert either flows to a filled ok — publish green, sleep, go again — or peels off to a faded alert the moment any step misbehaves. The linear spine is the happy path a customer would take; the branch is the entire point.

The honest tradeoffs

A canary is a probe, not a proof, and it’s worth being clear-eyed about the limits:

• It tests one path, once. A green canary means the synthetic tenant’s round-trip worked at that moment. A bug specific to one customer’s data shape sails right past it. It narrows “is anything broken?” from “we’ll find out when someone complains” to “within a few minutes” — a huge win, but not full coverage, and pretending otherwise is how you get complacent.
• False alarms erode trust faster than silence. Every design choice above — shape-not-value assertions, cleanup-first idempotency, the dedup window — protects the one thing that makes a canary useful: that when it goes red, people believe it. Spend that credibility on a flaky check and the apparatus is worthless.
• It costs real compute every run. A full onboard → seed → serve → tear-down cycle is real work against real services, on repeat, forever. The schedule interval is the knob: too often and you pay continuously to learn what you already know; too rarely and your detection window grows. You’re buying confidence with CPU.

The takeaway

Unit tests answer “do the parts work?” Health checks answer “is the server up?” Neither answers the question your customers actually care about: “does the whole thing still work end to end?” A synthetic canary answers that one directly, by being the customer — quietly, on a schedule, against a tenant that can’t hurt anyone. Build one the moment the cost of a silent integration failure is higher than the cost of a fake customer’s round-trip. For us, that math wasn’t close.