Chained Azure Functions for resumable, high-volume API sync

Here is a job that sounds simple and isn't: pull a large, paginated external API into your own database on a schedule. Tens of thousands of records. A strict rate limit. And a hard requirement that a failure halfway through never corrupts the data or restarts the whole run.

Why one big job fails

The obvious implementation is a single scheduled job that loops through every page. It fails for two reasons that arrive together.

First, the execution ceiling. Serverless functions have a maximum run time; a long sync runs straight into it and dies mid-flight. Second, the rate limit. Push the API as fast as one loop can and it starts returning 429s — so the same job is now both too slow to finish in its time budget and too fast for the API to tolerate. Those constraints don't compromise. They just both win, and the job loses.

The chain

The fix is to stop thinking of it as one job. Model it as a chain of small ones.

A timer triggers the first run. Each run does a bounded, predictable slice of work — one page range — and then, as its final act, invokes the next run over HTTP, handing it a resumption token: where to start, what the last successfully processed record id was. Each link in the chain is well inside the execution ceiling because each link is small by construction. The chain continues until there is nothing left to fetch.

Because progress is tracked by last-processed id, a failure costs exactly one link. The chain restarts from the resumption token, not from zero. The sync became resumable the moment it became incremental.

A job too big for one execution isn't a job that needs a bigger execution. It's a job that needs to be a chain.

Backoff that adapts

Inside each link, the rate limit still has to be respected — but a fixed sleep is a poor tool. Too short and you still get throttled; too long and the sync crawls. Adaptive backoff responds to what the API is actually telling you: slow down when it signals pressure, recover when it doesn't, and add jitter so retries don't synchronize into a thundering herd.

Land it safely

The last mile is the write. Records land through bulk upserts — set-based, not row-by-row — so a re-run of a link is idempotent: the same page processed twice produces the same database state, not duplicates. And where the source is nested JSON, mappers flatten those arrays into relational columns at write time, so what lands is something the rest of the business can actually query.

When to reach for Durable instead

This pattern is deliberately lightweight — no durable orchestration runtime, no extra infrastructure to operate. That is its appeal and its limit. If you need fan-out across hundreds of parallel branches with coordinated completion, or human-in-the-loop steps, or strict exactly-once guarantees across a complex graph, a durable orchestration framework earns its weight.

A paginated sync is none of those things. It is a straight line that must survive being interrupted. Chained functions with a resumption token give you exactly that — and nothing you have to babysit.