Skip to main content

Sitemaps and robots.txt for Small Sites

Technical basics · 8 min read ·

Two small files that guide search crawlers: what a sitemap and a robots file do, what they do not, and the mistakes that keep a site out of results.

Illustration: Two plain text file icons side by side labelled sitemap and robots with three short example lines each in monospace

Behind most small websites sit two modest text files that few owners ever look at: a sitemap and a robots file. One tells search engines what you would like them to find. The other tells them where they should not go. Both are simple, and both are easy to get wrong in ways that quietly reduce how well a site performs in search.

This guide explains what each file does, what it does not do and how to check them without being a developer.

The sitemap: a list of what you want found

Google describes a sitemap as a file that provides information about the pages, videos and other files on your site, and the relationships between them. Search engines read it to crawl your site more efficiently. It can carry extra details, such as when each page last changed, and information for specific kinds of content such as images and video.

Google lists when a sitemap helps most: when your site is large, when it is new with few external links pointing to it, or when it has a lot of rich media. A small, well-linked site can often be crawled fully without one, but a sitemap is cheap to provide and gives you a place to state exactly which pages matter.

What goes in it

  • The pages you want in search results: home, services, products, contact, about, articles.
  • The main address of each page, the one you want shown, not variations with tracking tags.
  • The last-modified date, if it is accurate.

What to leave out

  • Pages you have marked noindex. Listing them sends contradictory signals.
  • Duplicates and redirects. Include the final address only.
  • Pages that return errors.
  • Private or admin pages.
  • Thin pages of little value, such as empty tag pages.

How to create one

Most website platforms generate a sitemap automatically. Look for a setting or a plug-in, and check that it produces a file at a predictable address, often something like the site's address followed by a sitemap file name. Open it in a browser. You should see a tidy list of your pages. If you see hundreds of addresses you do not recognise, or none of your key pages, something is wrong.

How to tell search engines about it

Submit the address through the search engines' webmaster tools, such as Google Search Console, and, if you like, list it in your robots file so that any crawler can find it. Then monitor the report: it will show how many pages were found and any problems.

The robots file: instructions for crawlers

A robots file sits at the top level of a site and gives crawlers instructions about which addresses they may request. Google explains that it is mainly for managing crawler traffic, so that your site is not overloaded with requests, and for keeping certain files, such as media or unimportant scripts, out of crawlers' way.

What it is not for

Google is explicit about this. A robots file is not a mechanism for keeping a web page out of Google. A page blocked by robots can still appear in results, without a description, if other sites link to it. It is also not a security measure: crawlers that choose to ignore it can still request the page, and the file itself is public. For anything sensitive, use proper protection such as a password.

What it can do for a small site

  • Keep crawlers out of parts of the site that have no value in search, such as internal search results or basket pages.
  • Point crawlers to your sitemap.
  • Reduce server load if a crawler is requesting too much.

For many small sites, a minimal file that blocks nothing and points to the sitemap is all that is needed.

The danger

The most common serious mistake is a rule that blocks the whole site. This often happens when a site is built on a test version with a "block everything" rule, and the file is copied over to the live site at launch. The result is that crawlers are told to stay away from everything, and the site vanishes from search or never appears. Check your robots file after every launch and every major change.

Robots versus noindex

These two are often confused. They do different jobs.

  • Robots rules control whether a crawler may request a page.
  • A noindex instruction tells search engines not to include a page in results. It is set with a tag in the page or a header from the server.

Google explains the catch: for noindex to work, the page must not be blocked by the robots file, because the crawler has to fetch the page to see the instruction. If you block a page in robots and also add noindex, the crawler never sees the noindex, and the page may still appear in results because of links from elsewhere.

So, to keep a page out of search: allow crawling, add noindex. To keep a page private: protect it with a password.

Check yours in ten minutes

You do not need special tools.

  1. Open your robots file by adding its name to the end of your site's address. Read it. Is anything blocked? Does it block the whole site? Does it mention your sitemap?
  2. Open your sitemap the same way. Are the pages you care about listed? Are any unwanted ones?
  3. Pick three important pages and view their source, looking for a noindex tag. Or use the inspection tool in Search Console, which tells you whether a page can be indexed.
  4. Check the sitemap report in Search Console. Google describes this tool as helping you confirm that Google can find and crawl your site and fix indexing problems.
  5. Test on a phone. Crawling and indexing now focus on the mobile version.

If anything looks wrong, fix it or ask whoever built the site.

Keep them up to date

Sitemaps should change when your site does. If your platform generates them automatically, check that it updates when you add or remove pages. If you maintain one by hand, set a reminder. Recheck the robots file after any site rebuild, platform change or move to a new address.

Other crawlers

Search engines are not the only visitors. Other kinds of crawler, including those that collect data for assistants, may also read your robots file. Treat the file as a statement of what you are comfortable with, and remember that it is advisory. If you want to restrict particular crawlers, you can name them in the file, but this is a decision with consequences for how visible you are, so think it through.

A worked example

A small physiotherapy clinic launches a new site. A week later, its name does not appear in search. The owner checks the robots file and finds a single line blocking everything, left from the testing stage. She asks the developer to remove it, replaces it with a simple file that points to the sitemap and checks the sitemap, which lists twelve pages, including an old test page and a thank-you page that she does not want indexed.

She removes those two from the sitemap, adds noindex to the thank-you page, allows crawling of it and submits the sitemap in Search Console. A few days later, the report shows ten pages discovered. Over the following weeks, the home page and service pages appear in results. She adds a note to her launch checklist: "Check robots, noindex and sitemap on launch day."

Common mistakes

  1. A leftover block-everything rule.
  2. Blocking a page and expecting it to disappear from search.
  3. Noindex pages listed in the sitemap.
  4. A sitemap full of redirects and errors.
  5. A sitemap that is never updated.
  6. Treating robots as security.
  7. Not checking after a site change.

A checklist

  • Robots file reviewed, with no accidental block
  • Sitemap generated, accurate and submitted
  • Noindex used correctly and not blocked by robots
  • Private material protected properly
  • Checked after every launch and change
  • Search Console monitored

The search page here shows a minimal search, the categories page shows how listings are grouped and the submit page lets a business add a listing that links to its site.

When you change platform or address

Moves are the moments when these files cause the most damage. If you change your website platform, check that the new one generates a sitemap and a sensible robots file. If you move to a new address or domain, set up redirects from every old page to its new equivalent, update the sitemap with the new addresses and tell the search engines through their webmaster tools. Keep the old site's redirects in place for at least a year. Compare the number of indexed pages before and after, and investigate any big drop. A short checklist and a calm week usually prevent weeks of lost visibility.

Documenting your setup

Write a short note on where your sitemap and robots file live, who can edit them and what each is meant to do. Put it with your other site documentation. When a developer or agency changes hands, the note prevents accidental changes, and when something goes wrong, it tells you where to look first. A five-line note is enough, and it can save a great deal of time when a problem appears at an inconvenient moment.

Frequently asked questions

Do I need to name my sitemap in robots? It is not required, but it is a convenient place for crawlers to find it.

Can I have more than one sitemap? Yes, especially for larger sites, with an index file that lists them.

What if my platform does not let me edit these files? Many do it through settings. Check the platform's help pages or ask support.

Questions and answers

What does a sitemap do?
It lists the pages you want search engines to know about, along with details such as when they last changed, to help crawlers find them.
What does robots.txt do?
It tells crawlers which parts of a site they may request, mainly to manage crawler traffic. It is not a way to hide pages from search.
How do I keep a page out of search?
Use a noindex instruction, or protect the page with a password, and do not block it in robots.txt, because crawlers must be able to see the noindex.
Do small sites need these files?
A small, well-linked site may be found without a sitemap, but one is cheap insurance. A robots file is optional, but a mistake in one can do harm.

Sources

Related guides

Get help