Adding robots.txt and a sitemap

robots.txt: a file that blocks nothing

A robots.txt file sits at the root of a site and tells crawlers which paths they may fetch. Heliocite's has three lines:

• User-agent: * addresses every crawler.
• An empty Disallow: means nothing is off limits.
• A Sitemap: line gives the full address of the sitemap.

With no file at all, crawlers would assume the same thing: everything is allowed. The file is still useful because the Sitemap: line lets any crawler find the sitemap without being told about it separately.

A common misunderstanding: robots.txt controls crawling, not indexing. If you block a page and other sites link to it, search engines can still list the URL; they just can't read what's on it. To keep a page out of search results, let it be crawled and give it a noindex tag instead. Heliocite's 404 page does exactly that.

sitemap.xml: five URLs

A sitemap is an XML list of the pages you want search engines to know about. Heliocite's lists the homepage, the practices page, the blog, and the two posts. A few rules it follows:

• Only canonical URLs. Each address in the sitemap matches the page's canonical link exactly, trailing slash included. The 404 page isn't listed.
• Honest lastmod dates. Google uses the last-modified date only when it proves accurate over time, so it should change when the content does and not otherwise.
• No priority or changefreq. Google has said it ignores both, so they're left out.

On a five-page site where every page is linked from the homepage, crawlers would find everything without a sitemap. It earns its place in Search Console: once submitted, Search Console reports how many of the listed URLs are indexed and why the rest aren't. That's the measurement this blog depends on.

A sitemap doesn't guarantee anything gets indexed. It's a hint about what exists, not a request that will be granted.

Keeping it up to date

The sitemap is written by hand, like the rest of the site. Each new post now means three edits: the post itself, the blog lists, and a new entry in sitemap.xml. If that gets tedious, it will be a sign the site needs a small build step, and that will be a post too.

What's next

Connecting Google Search Console, submitting the sitemap, and recording the first baseline: what is indexed, and what isn't yet.