Chained Azure Functions for resumable, high-volume API sync

Here is a job that sounds simple and isn't: pull a large, paginated external API into your own database on a schedule. Tens of thousands of records. A strict rate limit. And a hard requirement that a failure halfway through never corrupts the data or restarts the whole run.

I've had a sync with exactly this shape in production for a while now — an external data provider's paginated API feeding a client's reporting warehouse, full refresh each cycle because the API offers no reliable delta or deletion feed. What follows is the pattern, and then what production taught me about its sharp edges.

Why one big job fails

The obvious implementation is a single scheduled job that loops through every page. It fails for two reasons that arrive together.

First, the execution ceiling — and it's worth being precise, because the numbers drive the design. On Azure's Consumption plan, a function gets 5 minutes by default and 10 at most. Flex Consumption, Premium, and Dedicated default to 30 minutes and can be configured "unbounded" — but unbounded comes with asterisks: a 60-minute grace period during scale-in, 10 minutes during platform updates. A multi-hour loop on any plan is a dice roll against infrastructure events. And anything HTTP-triggered has a harder wall: if it doesn't respond within about 230 seconds, the load balancer returns a 502 while your code keeps running — the caller sees failure, the work continues blind, and now nobody knows the true state.

Second, the rate limit. Push the API as fast as one loop can and it starts returning 429s — so the same job is now both too slow to finish in its time budget and too fast for the API to tolerate. Those constraints don't compromise. They just both win, and the job loses.

The chain

The fix is to stop thinking of it as one job. Model it as a chain of small ones.

A timer triggers the first run. Each run does a bounded, predictable slice of work — one page range — and then, as its final act, invokes the next run, handing it a resumption token: where to start, what the last successfully processed record id was. The token lives in a tracking row in the database, not in memory — it is the job's state. Each link in the chain is well inside the execution ceiling because each link is small by construction. The chain continues until there is nothing left to fetch.

Because progress is tracked by last-processed id, a failure costs exactly one link. The chain restarts from the resumption token, not from zero. The sync became resumable the moment it became incremental.

Why HTTP for the hand-off, when a queue was available? Honestly: simplicity. No extra infrastructure, and every link is independently invokable by URL — which makes "resume from record 45,000" a one-line curl during an incident instead of a message-crafting exercise. That choice has a real cost, which production eventually presented as an invoice; more on that below.

This is the same discipline Microsoft's own reliability guidance asks for: functions should be stateless and idempotent, and "if a previous run failed to complete, the next run should pick up where it left off".

A job too big for one execution isn't a job that needs a bigger execution. It's a job that needs to be a chain.

Backoff that adapts

Inside each link, the rate limit still has to be respected — but a fixed sleep is a poor tool. Too short and you still get throttled; too long and the sync crawls. Adaptive backoff responds to what the API is actually telling you: honor a Retry-After header when one is sent, back off exponentially when pressure continues, recover when it stops, and add jitter so parallel clients don't synchronize into a thundering herd. None of this is exotic — it's the standard transient-fault-handling guidance, applied inside a loop that also has a clock to watch.

Land it safely

The last mile is the write. Records land through bulk upserts — set-based, not row-by-row — so a re-run of a link is idempotent: the same page processed twice produces the same database state, not duplicates. Azure's docs treat designing for identical input as a first-class requirement for exactly this reason: retries, replays, and overlapping triggers will hand you the same data twice, and the write path has to shrug. And where the source is nested JSON, mappers flatten those arrays into relational columns at write time, so what lands is something the rest of the business can actually query.

What production taught me

The pattern as described worked. The edges needed sanding, and each lesson came from a real incident or a near miss during a hardening pass this year.

The hand-off is the weakest link. Invoking the next link over HTTP as a fire-and-forget call means one swallowed exception at the hand-off and the chain just... stops. Quietly. We had exactly one outage in this sync's life, and it was that: a continuation failure caught by a catch-all logger, a job that looked complete, a dataset that was silently short. Two fixes, both cheap: never wrap the continuation in the same catch-all as the work, and run a watchdog — a tiny timer function that alerts when a run started but never wrote its completion row. Better still, hand off through a queue message instead of an HTTP call; Microsoft's guidance recommends storage queues for cross-function communication precisely because a queue-triggered link gets retries and poison-message handling for free.

One cursor, one writer. A timer trigger, a manual kick-off, and an in-flight resume can all run at once — and if they share one tracking row, they corrupt each other's cursor and each other's data. The fix is a run lock: an in-progress flag with an expiry, taken before the first page and released at completion. Overlap isn't a rare event; it's what happens the first time someone re-triggers a "stuck" sync that isn't stuck.

Budget for the worst page, not the average. My links originally checked the clock at the top of each loop iteration against a nine-minute cap under a ten-minute ceiling. Fine on average — until one page cost a maximum rate-limit wait plus a full HTTP client timeout, and the check at the top of the loop couldn't save an iteration already in flight. Headroom has to be ceiling minus the worst possible single iteration, not ceiling minus a hopeful average.

Verify at the end. A chain that finishes isn't proof the data is whole — a skipped page or a short read completes just as cheerfully. So the final link runs a count probe against the source and, on mismatch, a census that back-fills only ids the source confirms are missing. (The first design flagged every absent integer between MIN and MAX as a gap; source ids aren't dense, and that "repair" would have invented work forever.) One more subtlety: verification writes must leave the resumption cursor untouched, or the repair pass corrupts the thing that makes the sync resumable.

The shape of a link, after all four lessons:

// One link: bounded work, durable cursor, safe hand-off
var budget = ceiling - worstCaseIteration;            // worst page, not average
await AcquireRunLockAsync();                          // one cursor, one writer
while (await HasMorePagesAsync(cursor) && sw.Elapsed < budget)
{
    var page = await FetchPageAsync(cursor);          // adaptive backoff + jitter inside
    await BulkUpsertAsync(page.Records);              // set-based, idempotent
    cursor = await SaveCursorAsync(page.LastId);      // the resumption token, persisted
}
if (await HasMorePagesAsync(cursor))
    await EnqueueNextLinkAsync(cursor);               // queue beats fire-and-forget HTTP
else
    await VerifyThenMarkCompleteAsync();              // the row the watchdog checks

When to reach for Durable instead

This pattern is deliberately lightweight — no durable orchestration runtime, no extra infrastructure to operate. That is its appeal and its limit. The managed version of this exact idea exists: function chaining is the first pattern in the Durable Functions catalog, with checkpointing and replay built in, and the operational cost of going durable dropped meaningfully once the Durable Task Scheduler went GA as a managed backend.

So here's the honest test, learned the hard way: every safeguard above — the run lock, the watchdog, the queue hand-off — is a small hand-rolled piece of what a durable runtime provides wholesale. One or two of them is normal hardening. If you find the list still growing, or you need fan-out across parallel branches, human-in-the-loop steps, or coordination across a complex graph, stop bolting and switch.

A paginated sync is none of those things. It is a straight line that must survive being interrupted. Chained functions with a resumption token — plus a lock, a watchdog, and a count check — give you exactly that, and nothing you have to babysit.