The generation token: verifying a dataset that races its own reload

A data load and the job that verifies it are usually written as if they take turns. Load finishes, verification starts, verification finishes, repeat. Real systems don't take turns. Rate limits stretch a verification pass into tens of minutes, schedules overlap, and sooner or later the next load fires while the last verification is still running. If you didn't design for that overlap, the result isn't a small glitch - it's a false alarm at full volume.

The collision

Picture a pipeline that truncates its target and reloads it - correctly, because the source API offers no trustworthy delta; that decision is its own post ("Full reload beats the delta you can't trust"). A separate verifier then crawls the source, confirming every record made it. The verifier is slow because the source is rate-limited to three requests a minute: a census of the full ID set plus any back-fill work runs tens of minutes to hours, split across self-chaining function invocations because a single Azure Function execution tops out at ten minutes. Meanwhile, reloads fire from a twice-monthly 3 a.m. timer - and from a manual HTTP trigger that any operator can hit whenever the data looks stale.

Halfway through a verification pass, a reload fires and truncates the table out from under it. Every check the verifier now makes finds nothing, because the table is mid-rebuild. It concludes, with total confidence, that the entire dataset is missing - and kicks off a back-fill storm against data that's actually fine and already being reloaded. At three requests a minute, the "correction" is hours of phantom fetches, all of it now colliding with the very load that's repairing the table.

The verifier wasn't wrong about what it saw. It was wrong about which load it was looking at.

Bind the work to a generation, not a clock

The fix is to give every load an identity and make the verifier loyal to one. Stamp an immutable LoadId - a GUID, a generation token - onto the sync-tracking row inside the same transaction that performs the truncate. The verifier captures the current LoadId when it starts, and before every continuation and every corrective write, it re-reads the token. If it changed, a newer load has superseded this one; the verifier marks its attempt Superseded and stops. It never writes against a generation it didn't begin with. The core loop, near-verbatim from the spec:

verify():
  loadId   = tracking.LoadId            # generation captured at start
  expected = probeSourceTotal()
  loop while work remains and elapsed < 9 min:
      if tracking.LoadId != loadId:     # a reload truncated under us
          finish(Superseded); return    # the new load triggers its own verify
      fetch next batch; upsert batch
  if work remains: self-chain the next invocation   # which re-checks again
  else: finish(Complete)

Two details carry the weight. Stamping inside the truncate transaction means there is no instant where the table is rebuilt but the token is stale. Checking before every side effect - not just at startup - means a verifier that has been running for forty minutes can't do damage in minute forty-one on the strength of a decision it made in minute one.

Long-running asynchronous work should be loyal to the generation it started with, not to wall-clock time. Stamp the generation at the source, check it before every side effect, and stand down when it moves.

This idea has names

I didn't invent this; I converged on it, which is usually a good sign. Martin Kleppmann's "How to do distributed locking" describes the fencing token: a monotonically increasing number issued with a lock, checked by the storage layer so that writes from a paused, stale lock-holder get rejected. The generation token is the same move with equality instead of ordering - the worker asks "is my generation still the current one?" before each write. What matters in both is where the check happens: at the moment of the side effect, not at the moment of acquisition.

The sync-tooling world converged on the same noun: Airbyte's refresh machinery stamps records with a generation ID precisely so that anything belonging to an older generation is recognizably stale rather than silently wrong. And at row scale, optimistic concurrency - ETags, rowversion columns - is the same discipline in miniature: read a version, write only if it hasn't moved.

It's worth saying why the obvious alternative loses. You could try to prevent the overlap with a lock, serializing loads and verifications. But the verifier legitimately runs for hours under the rate limit: blocking reloads behind it trades data freshness for verification hygiene, which is exactly backwards - and killing the verifier instead means verification never completes. Kleppmann's deeper point applies too: having held a lock at some earlier moment proves nothing about freshness at the moment you write; only a check at write time does. Overlap is fine and often unavoidable. The point isn't to prevent it - it's to make the stale worker recognize itself as stale before it does damage.

Make "stood down" a first-class outcome

An operational detail that earns its keep: Superseded is a terminal status, written durably to the verification history alongside Complete and Error - not a log line, not an exception. That does two things. It stops the stale attempt from being retried as if it had failed - the correct response to supersession is nothing, because the new load triggers its own verification. And it makes the overlap rate measurable: if every pass ends Superseded, verification never completes and the schedule itself is the bug - visible on a dashboard instead of buried in traces. A verifier that quietly dies when the table changes and one that records "a newer generation took over" look identical in the moment; they look very different a month later when someone asks which loads were ever actually verified.

Fire-and-forget needs a watchdog

The generation token kills the false alarm. It doesn't solve the other failure mode of these pipelines: the dropped hand-off. This chain continues by invoking its next link over HTTP, and the trigger helper treated a timeout on that call as "the function is likely running." Usually true - the next link started and just didn't respond quickly. Occasionally the next link never ran at all, and the chain stopped mid-stream with no error anywhere. On a twice-monthly schedule, "wait for the next scheduled run to self-heal" means up to two weeks of quietly stale data.

Persisting in-progress state - a row marked InProgress, a non-terminal status with a last-touched timestamp - makes recovery possible. It does not make recovery happen. Something has to notice the stalled generation and kick it. That something has a name too: in the Azure Architecture Center's Scheduler-Agent-Supervisor pattern, the supervisor is the component that periodically scans the durable state store for steps that have timed out or failed and arranges for them to be resumed. Here that's a small timer function: find non-terminal work whose last update is older than a threshold, re-trigger it. The spec reduced it to one acceptance criterion: recovery must not depend on the next scheduled sync.

(If I were building this chain fresh, I'd start from Durable Functions, whose orchestrations checkpoint progress durably and whose monitor pattern packages recurring poll-until-done work with managed lifetime - the platform's rendering of the same two ideas: durable state, plus something whose whole job is to wake up and look at it.)

The short version

Overlap between a load and its verifier is inevitable; design for it. Stamp an immutable generation token in the truncate transaction, have the verifier re-check it before every write and stand down - durably, as a first-class Superseded outcome - when it moves, and add a watchdog that actively resumes stalled non-terminal work. The pattern has pedigree: fencing tokens in distributed locking, generation IDs in sync tooling, the supervisor in Scheduler-Agent-Supervisor. The shared moral is the same everywhere: persisted state makes recovery possible, the watchdog makes it real, and the write-time check makes it safe.