Why a sitemap and robots.txt file still matter in 2020

It's a fair question, especially from a technically minded client: with search engines as sophisticated as they've become, do a sitemap and a robots.txt file still actually matter, or are they leftover relics from an earlier, less capable era of search technology. The honest answer is that they still matter meaningfully, for reasons that have shifted somewhat but haven't actually disappeared.

Both are small, unglamorous technical files, easy to overlook entirely, and both directly affect how thoroughly and how correctly a site gets crawled and understood by search engines. We check both on every audit we run, and we still find them missing or badly configured more often than a technically sophisticated client would probably expect.

An XML sitemap gives search engines a direct, explicit list of every page on a site that's meant to be indexed, along with useful metadata like when it was last meaningfully updated. For a small site with a simple, shallow structure, search engines can often discover most pages through normal crawling alone. For a larger or more complex site, a sitemap significantly speeds up and improves the completeness of that discovery process.

We generate and actively maintain a current sitemap for every client site, and we resubmit it whenever a site's structure changes meaningfully, because an outdated sitemap pointing at pages that no longer exist, or missing pages that are genuinely new, actively works against the exact purpose it's meant to serve.

This file tells search engine crawlers which parts of a site they're allowed to access and which parts they should skip entirely, admin areas, duplicate content generated automatically by certain site features, staging environments that were accidentally left publicly accessible. Getting this file wrong can accidentally block search engines from an entire site, or fail to block content that genuinely shouldn't be indexed at all.

We've inherited more than one site where a misconfigured robots.txt file was silently blocking search engines from crawling meaningful sections of otherwise perfectly good content, invisible to a human visitor browsing normally, and invisible to the site owner as well, until someone specifically checked this one small, easy-to-overlook file directly.

A single misplaced rule in robots.txt can accidentally block an entire site from being crawled at all, a mistake we've seen happen after a site migration when a setting meant only for a temporary staging environment accidentally carried over into the live production site without anyone specifically checking for it before launch.

The damage from this specific kind of mistake compounds the longer it goes unnoticed, since search engines may take real time to fully recrawl and correctly re-index a site once a blocking rule like this is finally identified and properly removed, meaning a mistake caught late can cost real, meaningful ranking recovery time on top of whatever visibility was already lost while the block was silently in place.

We manually review the actual robots.txt file content on every new client site, checking for both overly broad blocking rules and any obviously missing rules that should genuinely be there to protect non-public areas. We also validate the sitemap against the site's actual current structure, confirming it's both complete and genuinely current rather than a stale artifact left over from an earlier version of the site.

This check takes minutes and has caught real, meaningful problems on a surprising share of sites we've audited, including some belonging to businesses that had a technically sophisticated internal team who, understandably, simply hadn't thought to check these two specific, unglamorous files directly and deliberately.

Search engines have genuinely gotten better at discovering and understanding site structure without needing either file present at all, which is part of why some assume these files no longer matter much in practice. They still meaningfully improve crawl efficiency and prevent real, specific mistakes, which is why we treat them as basic due diligence rather than optional legacy technical debt from an earlier, less capable era.

We frame this clearly to technically sophisticated clients who question whether these older mechanisms are still worth the attention: think of them less as essential in isolation, and more as cheap, low-effort insurance against a mistake that could otherwise cost real, measurable visibility for weeks or months before anyone happens to notice it.

Add "/robots.txt" directly to your own domain in a browser and actually read what comes back. If it says "Disallow: /" at the top level, or something close to that, your entire site may be blocked from search engines right now, which is a far more common and far more costly mistake than most site owners would ever expect to find on their own site.

We recommend every business owner check this directly, today, regardless of whether they're actively working with an SEO agency or handling everything entirely in-house, because it takes thirty seconds and it's genuinely one of the highest-consequence checks available for so little effort involved.

Site migrations and redesigns are, by a wide margin, the single riskiest moment for both of these files, since a staging environment's protective robots.txt rule is exactly the kind of setting that's easy to forget to update when a new site actually goes live in its place. We now include an explicit robots.txt and sitemap check as a mandatory final step in every launch checklist we run, specifically because of how often this exact mistake occurs during precisely this kind of transition.

This single addition to our launch checklist has prevented what would otherwise have been a genuinely serious, if entirely avoidable, mistake on at least a couple of projects since we started requiring it as a non-negotiable final step before any site goes live.

These two files are small, unglamorous, and easy to forget entirely, and they still directly affect how completely and how correctly a site actually gets crawled and understood by search engines in 2020.

We check both on every project, every time, because the cost of checking is minutes, and the cost of an undetected mistake here can be weeks or months of avoidable lost visibility, entirely preventable with a quick, deliberate look.

If you've never personally checked either file on your own site, that's a five-minute task worth doing today, before assuming everything is configured correctly by default just because nothing looks obviously broken to a normal visitor browsing the site.