Robots.txt Generator for WordPress

What Is a Robots.txt File?

A robots.txt file is a small, plain-text file that lives at the root of a website — accessible at yourdomain.com/robots.txt — and tells search engine crawlers which parts of the site they're allowed to visit and which parts they should skip. It's one of the first files a search engine bot checks before crawling a site, and while it doesn't guarantee a page won't ever be indexed, it does control crawl behavior in a way that affects how efficiently search engines spend their time on a site.

The file follows the Robots Exclusion Protocol, a simple text-based standard supported by all major search engines including Google, Bing, and others. It's made up of one or more 'User-agent' blocks, each specifying which crawler the rules apply to, followed by 'Allow' and 'Disallow' directives that define which paths that crawler can or can't access.

Why WordPress Sites Specifically Need One

WordPress generates a huge number of URLs that aren't meant for search engines — admin pages, login screens, plugin asset folders, internal search result pages, and cart or checkout pages on WooCommerce stores. Without a properly configured robots.txt, search engines can waste crawl budget indexing these irrelevant pages instead of focusing on the actual content that matters, like service pages, blog posts, or product listings.

WordPress does generate a very basic virtual robots.txt by default if no physical file exists, but it's minimal and doesn't account for the specific structure of a given site — its plugins, custom post types, or e-commerce setup. A tailored robots.txt, built for the actual site rather than a generic default, gives far more control over crawl behavior.

Common Directives Explained

User-agent specifies which crawler the following rules apply to — an asterisk (*) means the rules apply to all crawlers, while a specific name like Googlebot targets only Google's crawler. Disallow tells crawlers not to access a specific path, while Allow explicitly permits access to a path, which is useful for allowing a specific file inside an otherwise disallowed folder (a common example being admin-ajax.php inside the disallowed /wp-admin/ folder, since many WordPress features rely on that file being reachable).

The Sitemap directive points crawlers to the site's XML sitemap, helping them discover all the important pages efficiently rather than relying purely on following internal links. Crawl-delay, supported by some crawlers (though notably not Google), tells bots to wait a specified number of seconds between requests — useful for sites on limited server resources that could be slowed down by aggressive crawling.

A Note on AI Crawlers

As AI systems increasingly crawl the web to train language models, some site owners choose to block specific AI-related user agents — like GPTBot (OpenAI), CCBot (Common Crawl), and Google-Extended (Google's AI training crawler, separate from regular Googlebot) — if they don't want their content used for AI training purposes. This is a business decision rather than a technical requirement; blocking these crawlers doesn't affect regular search engine indexing or ranking, since they're distinct from the crawlers that power search results.

Common Mistakes to Avoid

Accidentally disallowing the entire site with 'Disallow: /' under 'User-agent: *' is the most damaging mistake — this tells every search engine to skip the whole site, which can happen if a staging-site robots.txt gets pushed to production by accident. Another common issue is disallowing CSS or JavaScript files needed for Google to properly render and understand the page, which can hurt how the page is evaluated for search ranking. Blocking the sitemap itself, or forgetting to update the sitemap URL after a domain change, are smaller but still avoidable errors.

It's also worth remembering that Disallow doesn't guarantee a page stays out of Google's index — if other sites link to a disallowed page, Google may still index the URL itself (without crawling its content) since the link signals the page exists. For pages that must never appear in search results, a noindex meta tag on the page itself is the correct tool, not robots.txt alone.

Using This Generator

This tool builds a working robots.txt file based on the choices above — pre-filled with the WordPress paths most sites want blocked (wp-admin, wp-includes), with the option to add store-specific paths like /cart/ or /checkout/, block AI training crawlers, and automatically append the correct sitemap reference. Once generated, download the file and upload it to the root directory of the website (not inside a subfolder) so it's accessible at yourdomain.com/robots.txt exactly as search engines expect.

Testing Your Robots.txt After Upload

Once the file is live, it's worth confirming it actually works as intended rather than assuming it does. Google Search Console includes a robots.txt report under the Settings section that shows the last time Google fetched the file and flags any syntax errors it found. Manually visiting yourdomain.com/robots.txt in a browser is also a simple sanity check — the file should load as plain text, not trigger a 404, and not be wrapped in any theme or page template.

It's worth re-checking the file after any major site change — a new WooCommerce store added to an existing site, a membership area introduced, or a migration to new hosting — since these changes often introduce new paths that either should or shouldn't be crawled, and the original robots.txt won't automatically account for them.

Robots.txt vs Meta Robots Tags vs Rank Math Settings

It's easy to confuse robots.txt with the meta robots tag (the 'index, follow' or 'noindex, nofollow' tag added inside a page's head section) and with the per-page visibility settings inside SEO plugins like Rank Math or Yoast. Robots.txt operates at the crawl level — it tells bots which URLs to request in the first place. The meta robots tag operates at the indexing level, on a specific page a bot has already been allowed to crawl, and directly controls whether that page can appear in search results. Rank Math and Yoast's per-page settings are simply a friendly interface for editing that same meta robots tag without touching code directly.

Understanding this difference matters because disallowing a page in robots.txt does not remove it from search results if it's already indexed or linked from elsewhere — a noindex tag is the correct tool for that. The two systems work together: robots.txt manages what gets crawled, while meta robots tags manage what gets shown in search results.