Generate your sitemap from the database, not by hand

A hand-maintained sitemap.xml is drift waiting to happen. You publish an article, you forget the sitemap, and search engines never learn the page exists. The file and the reality it's meant to describe diverge in the first week, and nobody notices, because nothing errors.

This site's version of that failure was purer: an SEO audit in June found sitemap.xml returning 404. There was no file to drift — nobody had ever remembered to make one, which is the same disease at an earlier stage.

The fix was not to write the file. The fix was to stop it being a file: a small controller now builds the sitemap from the same database the pages render from — static routes first, then every capability, case study, and article, with one Where clause doing all the governance:

.Where(a => a.Status == ContentStatus.Published)

Every published item in, every draft out, automatically. It can't fall out of sync with the content because it is the content, queried at request time and cached for an hour. Publish a post and it's in the sitemap; unpublish it and it's gone — no second step to forget. It went live serving 38 URLs, and every article published since has appeared on its own, because there is nothing to remember.

Deriving from the database also makes <lastmod> honest for free: it's each row's own UpdatedUtc, stamped when the content last changed — which happens to be the standard Google holds the tag to: it's used only when "consistently and verifiably accurate". A hand-edited date rarely clears that bar; a derived one can't miss it. (Two honest footnotes from that same page: Google ignores priority and changefreq, so don't bother emitting them — and a small, well-linked site may not strictly need a sitemap at all. The win here isn't a ranking trick; it's deleting a manual sync step forever.)

robots.txt got the same treatment — emitted from code, host-aware, ending with the Sitemap: pointer the sitemaps.org protocol defines — so the crawl rules ship with the application instead of living beside it. The only genuinely manual step left is the one that lives outside the site: registering the sitemap with Search Console, once.

The principle outlasts the specific files: any artifact whose job is to mirror your content should be derived from that content, not maintained beside it. Two copies of the truth is one copy too many.