AutoSocialSyncGuides Reviews & Tools
HomeFeaturesToolsAI Text DetectorText to VoiceCharacter CounterAI ChatGuidesReviewsNewsResourcesAbout UsInternet Speed Test
Sign InSign Up
Blog / Guides

Robots.txt & Meta Tag Generator: How to Create robots.txt, Meta Robots and Open Graph Tags (2026)

Robots.txt & Meta Tag Generator: How to Create robots.txt, Meta Robots and Open Graph Tags (2026)

Two small pieces of code decide a lot about how search engines treat your website: the robots.txt file and the meta robots tag. A wrong line in either one can hide your best pages from Google or leave private pages open to crawlers. This guide explains what each one does, how to build both with our free generator, and how to avoid the mistakes that cost websites traffic.

Open the free Robots.txt & Meta Tag Generator

What is a robots.txt file?

A robots.txt file is a plain text file at the root of your website, for example https://www.example.com/robots.txt. It tells crawlers which URLs they may or may not request. Reputable search engines follow it, but it is a set of polite instructions and not a security feature. It is also public, so never list a path in it that you want to keep secret.

What is a meta robots tag?

A meta robots tag is a line of HTML in the <head> of a single page. It tells search engines whether to index that page, follow its links and how to show it in results. Where robots.txt works at the level of crawling, the meta robots tag works at the level of one page.

Robots.txt vs meta robots: the difference that matters

robots.txtMeta robots tag
Where it livesOne file in your site rootIn the head of each page
What it controlsWhether crawlers may request a URLWhether a page is indexed and how it appears
Can keep a page out of search?No, not reliablyYes, with noindex
Best forSaving crawl time, blocking admin or cart areasHiding thin, private or duplicate pages from results

Why Disallow does not remove a page from Google

This is the most common misunderstanding. A page blocked in robots.txt can still appear in search results if other websites link to it. To keep a page out of search, add a noindex meta robots tag (or an X-Robots-Tag HTTP header for files like PDFs) and do not block that page in robots.txt. If it is blocked, crawlers never see the noindex instruction.

How to use the Robots.txt & Meta Tag Generator

  1. Open the generator. Everything is generated in your browser, so nothing you type is uploaded. You can press Load sample to see a finished example.
  2. Choose the Robots.txt tab or the Meta Tags tab. The code on the right updates live with syntax colours as you type.
  3. Fill in the options below, check the SEO checklist and fix anything marked red or yellow.
  4. Press Copy code or download the files.

Building your robots.txt step by step

1. Website and sitemap

Enter your website URL and the full address of your XML sitemap, for example https://www.example.com/sitemap.xml. You can add more than one sitemap. A Sitemap: line helps search engines find your pages faster. The sitemap address must start with https://, and the tool warns you if it points to a different host than your website.

2. Who can crawl what?

Pick the crawler the rules apply to: all crawlers (*), Googlebot, Bingbot or a custom user-agent. Then choose a permission level:

  • Allow all โ€“ everything can be crawled. This is the right choice for most small sites.
  • Custom rules โ€“ block or allow specific paths.
  • Block all โ€“ nothing can be crawled. Use it only for a site under construction, because it removes your site from search over time.

3. Allow and Disallow rules

With custom rules you can add as many Allow and Disallow lines as you need. Use the Quick add buttons for common paths such as /wp-admin/, /admin/, /login, /cart/, /checkout/, /account/, /search, /tmp/ and /cgi-bin/. To block a whole file type, type extensions such as pdf, zip, docx and each becomes a rule like /*.pdf$. The $ symbol only works at the very end of a pattern, and paths cannot contain spaces.

4. Extra crawler groups

Give Googlebot, Bingbot or any custom crawler its own set of rules. Everything else follows your default group. Remember that an extra group does not inherit the rules of the default group, so repeat any rule it should also follow.

5. Block AI training crawlers

One switch adds rules for GPTBot, Google-Extended, CCBot, ClaudeBot, anthropic-ai, Applebot-Extended, Meta-ExternalAgent, Bytespider and cohere-ai. This only affects crawlers that honour robots.txt, so it is a request and not a guarantee.

6. Advanced directives

  • Crawl-delay asks a bot to wait a number of seconds between requests. Bing and some other crawlers follow it, but Google ignores it. To slow Google down, use the crawl settings in Search Console.
  • Host is a legacy directive that only Yandex ever used. Google and Bing ignore it, so it is better left empty.
  • Custom directives let you paste your own lines. Add a User-agent line before any custom Allow or Disallow rules.

7. Test a URL before you publish

Type a page, file path or full URL, choose a crawler such as Googlebot or Bingbot, and the tester tells you whether the URL would be allowed or blocked by your rules. Test your most important pages first.

Example of a finished robots.txt

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml

Creating your meta tags

Switch to the Meta Tags tab to build the code for the head of your page.

Basic SEO tags

  • Page title โ€“ aim for roughly 30 to 60 characters and put the important words first. A counter shows how close you are to 60.
  • Meta description โ€“ about 70 to 160 characters that explain what the page offers. A good description can improve click-through from search results.
  • Canonical URL โ€“ the main address of the page, which helps avoid duplicate-content confusion. It must be a full URL starting with https://.
  • Author, publisher, language, theme colour, encoding and viewport are optional extras.
  • Meta keywords is optional and largely obsolete. Google does not use it, so you can leave it empty.

Robots meta directives

DirectiveWhat it does
index / noindexAllow or prevent the page from appearing in search results
follow / nofollowAllow or prevent crawlers from following the links on the page
noarchiveNo cached copy of the page
nosnippetNo text snippet in search results
noimageindexDo not index the images on the page
notranslateDo not offer a translation of the page
max-snippetLimit the snippet length in characters (0 blocks it, -1 means no limit)
max-image-previewSet the largest image preview: none, standard or large
max-video-previewLimit video previews in seconds

Index and follow are the defaults, so the tool leaves the tag out unless you change something. You can switch on Always output the robots tag if you want it to appear anyway. Tip: max-image-preview:large makes your page eligible for large thumbnails in Google Discover and some results.

Open Graph and X Card tags

These tags control how your link looks when someone shares it on Facebook, LinkedIn or X. Set the title, description, image URL, type and site name. By default the tool reuses your page title, description and URL, so blank fields are filled for you. For the image, use an absolute URL, ideally 1200 ร— 630 pixels. For X, choose between summary_large_image and summary. A live search and share preview shows how your result and your shared card will look.

What the SEO checklist catches

The checklist marks problems in red, tips in yellow and passed checks in green, each with a plain-English explanation. Typical things it finds:

WarningWhy it matters
Your whole site is blockedDisallow: / hides everything from that crawler. Make sure it is intended.
Conflicting Allow and Disallow rulesThe same path is both allowed and blocked, which makes the result unclear.
CSS or JavaScript may be blockedGoogle needs them to render pages the way visitors see them.
Noindex together with a canonical URLThe two send mixed signals. Remove one of them.
Noindex page blocked in robots.txtCrawlers will never see the noindex tag.
Meta description missing or too longSearch engines pick random text, or cut yours off after about 160 characters.
Extra groups do not inherit default rulesA Googlebot group follows only its own lines, so repeat shared rules.
Crawl-delay set for GoogleGooglebot ignores Crawl-delay.

WordPress users also get a tip to allow /wp-admin/admin-ajax.php, because many themes and plugins load content through it.

How to put your files on your website

  1. Upload robots.txt. Save it as robots.txt (lowercase, UTF-8 plain text) and upload it to your site root so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
  2. Test it. Use the tester in the generator, then open the robots.txt report in Google Search Console and confirm your key pages are not blocked.
  3. Add the meta tags. Paste them inside the <head> of each page. Use a different title, description and canonical URL for every page.
  4. Recheck after changes. Crawlers cache robots.txt, so changes can take a while to apply. Review the file whenever you add sections or move pages.

5 robots.txt mistakes to avoid

  1. Leaving Disallow: / on a live site. It is often left over from development and can remove your site from search.
  2. Using Disallow to hide a page from Google. Use noindex instead, and leave the page crawlable.
  3. Blocking CSS and JavaScript files. Google may not understand your pages without them.
  4. Listing secret URLs. The file is public, so it shows attackers exactly where to look.
  5. Forgetting the sitemap line. It is one easy line that helps crawlers discover your pages.

Create your files now

Ready to build yours? Open the free Robots.txt & Meta Tag Generator, choose your rules, check the SEO checklist and download your files. You can also explore more free utilities on our tools page.

Robots.txt and meta tag FAQ

What is a robots.txt file?

It is a plain text file at the root of your website that tells crawlers which URLs they may or may not request. Reputable search engines follow it, but it is not a security feature.

Does Disallow remove a page from Google?

No. Robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it. Use a noindex meta robots tag or an X-Robots-Tag header, and do not block the page in robots.txt so crawlers can see the instruction.

Where do I put the robots.txt file?

Upload it to the root of each host, for example https://www.example.com/robots.txt. Subdomains and different protocols need their own file. It must be named robots.txt in lowercase and saved as UTF-8 plain text.

Do I still need the meta keywords tag?

No. Google does not use the keywords meta tag, and long keyword lists can look spammy. The generator keeps the field optional so you can leave it empty.

Can I block AI crawlers with robots.txt?

You can ask them to stay away, and the generator adds the common ones with one switch. Only crawlers that honour robots.txt will comply, so it is not a guarantee.

Does Google support Crawl-delay or the Host directive?

No. Googlebot ignores Crawl-delay, and Host was only ever used by Yandex and is now deprecated. Use Search Console for crawl settings, and redirects plus canonical URLs to choose your preferred domain.

What is the difference between noindex and nofollow?

Noindex asks search engines not to show the page in results. Nofollow asks them not to follow the links on the page. You can use them together or separately.

Is my data sent anywhere?

No. Everything is generated in your browser. Your settings are saved only on your own device so you can come back to them, and the Reset button clears them.

Is the generator free?

Yes. It is free to use with no sign-up.

โ† Back to Blog