Two small pieces of code decide a lot about how search engines treat your website: the robots.txt file and the meta robots tag. A wrong line in either one can hide your best pages from Google or leave private pages open to crawlers. This guide explains what each one does, how to build both with our free generator, and how to avoid the mistakes that cost websites traffic.
Open the free Robots.txt & Meta Tag Generator
What is a robots.txt file?
A robots.txt file is a plain text file at the root of your website, for example https://www.example.com/robots.txt. It tells crawlers which URLs they may or may not request. Reputable search engines follow it, but it is a set of polite instructions and not a security feature. It is also public, so never list a path in it that you want to keep secret.
What is a meta robots tag?
A meta robots tag is a line of HTML in the <head> of a single page. It tells search engines whether to index that page, follow its links and how to show it in results. Where robots.txt works at the level of crawling, the meta robots tag works at the level of one page.
Robots.txt vs meta robots: the difference that matters
| robots.txt | Meta robots tag | |
|---|---|---|
| Where it lives | One file in your site root | In the head of each page |
| What it controls | Whether crawlers may request a URL | Whether a page is indexed and how it appears |
| Can keep a page out of search? | No, not reliably | Yes, with noindex |
| Best for | Saving crawl time, blocking admin or cart areas | Hiding thin, private or duplicate pages from results |
Why Disallow does not remove a page from Google
This is the most common misunderstanding. A page blocked in robots.txt can still appear in search results if other websites link to it. To keep a page out of search, add a noindex meta robots tag (or an X-Robots-Tag HTTP header for files like PDFs) and do not block that page in robots.txt. If it is blocked, crawlers never see the noindex instruction.
How to use the Robots.txt & Meta Tag Generator
- Open the generator. Everything is generated in your browser, so nothing you type is uploaded. You can press Load sample to see a finished example.
- Choose the Robots.txt tab or the Meta Tags tab. The code on the right updates live with syntax colours as you type.
- Fill in the options below, check the SEO checklist and fix anything marked red or yellow.
- Press Copy code or download the files.
Building your robots.txt step by step
1. Website and sitemap
Enter your website URL and the full address of your XML sitemap, for example https://www.example.com/sitemap.xml. You can add more than one sitemap. A Sitemap: line helps search engines find your pages faster. The sitemap address must start with https://, and the tool warns you if it points to a different host than your website.
2. Who can crawl what?
Pick the crawler the rules apply to: all crawlers (*), Googlebot, Bingbot or a custom user-agent. Then choose a permission level:
- Allow all โ everything can be crawled. This is the right choice for most small sites.
- Custom rules โ block or allow specific paths.
- Block all โ nothing can be crawled. Use it only for a site under construction, because it removes your site from search over time.
3. Allow and Disallow rules
With custom rules you can add as many Allow and Disallow lines as you need. Use the Quick add buttons for common paths such as /wp-admin/, /admin/, /login, /cart/, /checkout/, /account/, /search, /tmp/ and /cgi-bin/. To block a whole file type, type extensions such as pdf, zip, docx and each becomes a rule like /*.pdf$. The $ symbol only works at the very end of a pattern, and paths cannot contain spaces.
4. Extra crawler groups
Give Googlebot, Bingbot or any custom crawler its own set of rules. Everything else follows your default group. Remember that an extra group does not inherit the rules of the default group, so repeat any rule it should also follow.
5. Block AI training crawlers
One switch adds rules for GPTBot, Google-Extended, CCBot, ClaudeBot, anthropic-ai, Applebot-Extended, Meta-ExternalAgent, Bytespider and cohere-ai. This only affects crawlers that honour robots.txt, so it is a request and not a guarantee.
6. Advanced directives
- Crawl-delay asks a bot to wait a number of seconds between requests. Bing and some other crawlers follow it, but Google ignores it. To slow Google down, use the crawl settings in Search Console.
- Host is a legacy directive that only Yandex ever used. Google and Bing ignore it, so it is better left empty.
- Custom directives let you paste your own lines. Add a
User-agentline before any custom Allow or Disallow rules.
7. Test a URL before you publish
Type a page, file path or full URL, choose a crawler such as Googlebot or Bingbot, and the tester tells you whether the URL would be allowed or blocked by your rules. Test your most important pages first.
Example of a finished robots.txt
User-agent: *
Disallow: /admin/
Allow: /admin/help/
Sitemap: https://example.com/sitemap.xml
Creating your meta tags
Switch to the Meta Tags tab to build the code for the head of your page.
Basic SEO tags
- Page title โ aim for roughly 30 to 60 characters and put the important words first. A counter shows how close you are to 60.
- Meta description โ about 70 to 160 characters that explain what the page offers. A good description can improve click-through from search results.
- Canonical URL โ the main address of the page, which helps avoid duplicate-content confusion. It must be a full URL starting with
https://. - Author, publisher, language, theme colour, encoding and viewport are optional extras.
- Meta keywords is optional and largely obsolete. Google does not use it, so you can leave it empty.
Robots meta directives
| Directive | What it does |
|---|---|
index / noindex | Allow or prevent the page from appearing in search results |
follow / nofollow | Allow or prevent crawlers from following the links on the page |
noarchive | No cached copy of the page |
nosnippet | No text snippet in search results |
noimageindex | Do not index the images on the page |
notranslate | Do not offer a translation of the page |
max-snippet | Limit the snippet length in characters (0 blocks it, -1 means no limit) |
max-image-preview | Set the largest image preview: none, standard or large |
max-video-preview | Limit video previews in seconds |
Index and follow are the defaults, so the tool leaves the tag out unless you change something. You can switch on Always output the robots tag if you want it to appear anyway. Tip: max-image-preview:large makes your page eligible for large thumbnails in Google Discover and some results.
Open Graph and X Card tags
These tags control how your link looks when someone shares it on Facebook, LinkedIn or X. Set the title, description, image URL, type and site name. By default the tool reuses your page title, description and URL, so blank fields are filled for you. For the image, use an absolute URL, ideally 1200 ร 630 pixels. For X, choose between summary_large_image and summary. A live search and share preview shows how your result and your shared card will look.
What the SEO checklist catches
The checklist marks problems in red, tips in yellow and passed checks in green, each with a plain-English explanation. Typical things it finds:
| Warning | Why it matters |
|---|---|
| Your whole site is blocked | Disallow: / hides everything from that crawler. Make sure it is intended. |
| Conflicting Allow and Disallow rules | The same path is both allowed and blocked, which makes the result unclear. |
| CSS or JavaScript may be blocked | Google needs them to render pages the way visitors see them. |
| Noindex together with a canonical URL | The two send mixed signals. Remove one of them. |
| Noindex page blocked in robots.txt | Crawlers will never see the noindex tag. |
| Meta description missing or too long | Search engines pick random text, or cut yours off after about 160 characters. |
| Extra groups do not inherit default rules | A Googlebot group follows only its own lines, so repeat shared rules. |
| Crawl-delay set for Google | Googlebot ignores Crawl-delay. |
WordPress users also get a tip to allow /wp-admin/admin-ajax.php, because many themes and plugins load content through it.
How to put your files on your website
- Upload robots.txt. Save it as
robots.txt(lowercase, UTF-8 plain text) and upload it to your site root so it opens athttps://yourdomain.com/robots.txt. Each subdomain needs its own file. - Test it. Use the tester in the generator, then open the robots.txt report in Google Search Console and confirm your key pages are not blocked.
- Add the meta tags. Paste them inside the
<head>of each page. Use a different title, description and canonical URL for every page. - Recheck after changes. Crawlers cache robots.txt, so changes can take a while to apply. Review the file whenever you add sections or move pages.
5 robots.txt mistakes to avoid
- Leaving
Disallow: /on a live site. It is often left over from development and can remove your site from search. - Using Disallow to hide a page from Google. Use
noindexinstead, and leave the page crawlable. - Blocking CSS and JavaScript files. Google may not understand your pages without them.
- Listing secret URLs. The file is public, so it shows attackers exactly where to look.
- Forgetting the sitemap line. It is one easy line that helps crawlers discover your pages.
Create your files now
Ready to build yours? Open the free Robots.txt & Meta Tag Generator, choose your rules, check the SEO checklist and download your files. You can also explore more free utilities on our tools page.
Robots.txt and meta tag FAQ
What is a robots.txt file?
It is a plain text file at the root of your website that tells crawlers which URLs they may or may not request. Reputable search engines follow it, but it is not a security feature.
Does Disallow remove a page from Google?
No. Robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it. Use a noindex meta robots tag or an X-Robots-Tag header, and do not block the page in robots.txt so crawlers can see the instruction.
Where do I put the robots.txt file?
Upload it to the root of each host, for example https://www.example.com/robots.txt. Subdomains and different protocols need their own file. It must be named robots.txt in lowercase and saved as UTF-8 plain text.
Do I still need the meta keywords tag?
No. Google does not use the keywords meta tag, and long keyword lists can look spammy. The generator keeps the field optional so you can leave it empty.
Can I block AI crawlers with robots.txt?
You can ask them to stay away, and the generator adds the common ones with one switch. Only crawlers that honour robots.txt will comply, so it is not a guarantee.
Does Google support Crawl-delay or the Host directive?
No. Googlebot ignores Crawl-delay, and Host was only ever used by Yandex and is now deprecated. Use Search Console for crawl settings, and redirects plus canonical URLs to choose your preferred domain.
What is the difference between noindex and nofollow?
Noindex asks search engines not to show the page in results. Nofollow asks them not to follow the links on the page. You can use them together or separately.
Is my data sent anywhere?
No. Everything is generated in your browser. Your settings are saved only on your own device so you can come back to them, and the Reset button clears them.
Is the generator free?
Yes. It is free to use with no sign-up.
