AutoSocialSync Guides Reviews & Tools
Home Features Tools AI Text Detector Text to Voice Character Counter AI Chat Guides Reviews News Resources About Us Internet Speed Test
Sign In Sign Up
Free SEO tool Β· Runs in your browser Β· No sign-up

Robots.txt & Meta Tag Generator

Choose which pages crawlers can visit and how search engines should treat your content. Clean, standards-aware code appears instantly as you type β€” no coding needed.

⚑ Start generating
  • ⚑ Live code preview
  • πŸ§ͺ URL tester
  • βœ… SEO checklist
  • 🌐 Open Graph & X cards
  • ⬇ Download files
⚑

Instant live preview

Every change updates your robots.txt and meta tags immediately, with syntax highlighting.

βœ…

Smart SEO checklist

Green, yellow and red checks explain conflicts, risky rules and missing tags in plain words.

πŸ§ͺ

Test any URL

Check whether a page or file would be allowed or blocked for Googlebot, Bingbot or any crawler.

πŸ€–

Multiple crawler groups

Set different rules for Googlebot, Bingbot or custom bots, or block common AI crawlers in one click.

🌐

Social & search tags

Open Graph and X Card tags with search and share previews, reusing your title and description.

πŸ”’

Private by default

Everything is generated in your browser. Nothing you type is uploaded.

Robots.txt and meta tag generator

1 Website & sitemap

Sitemap URL i

2 Who can crawl what?

Crawler permissions

3 Extra crawler groups

Give specific crawlers their own rules. Everything else follows the default group above.

βš™οΈ Advanced directives

Only Yandex ever used this, and it is deprecated. Google and Bing ignore it.

Placed after your groups. Include a User-agent line before any custom Allow or Disallow rules.

πŸ§ͺ Test a URL against these rules

Enter a path to see whether it would be crawled.
0

SEO validation

Checking your configuration…

How to use your files

From generated code to a live website in four steps

  1. Upload robots.txt

    Save the file as robots.txt and upload it to your site root so it opens at https://yourdomain.com/robots.txt.

  2. Test it

    Open the robots.txt report in Google Search Console and check that key pages are not blocked. Use the URL tester above first.

  3. Add the meta tags

    Paste the meta tags inside the <head> of each page. Use a different title, description and canonical URL for every page.

  4. Recheck after changes

    Crawlers cache robots.txt, so changes can take a while to apply. Review the file whenever you add sections or move pages.

Robots.txt does not guarantee deindexing

Robots.txt controls crawler access, not whether a URL is indexed. A blocked page can still appear in results if other sites link to it. To remove a page from search, add a noindex robots meta tag in the page HTML, or send an X-Robots-Tag HTTP header for files such as PDFs. A page blocked in robots.txt may never be crawled, so crawlers cannot discover its noindex instruction. Also remember that robots.txt is public: never list paths you want to keep secret.

FAQ

Questions, answered

What is a robots.txt file?

A robots.txt file is a plain text file at the root of your website that tells crawlers which URLs they may or may not request. It is a set of polite instructions that reputable search engines follow, not a security feature.

Does Disallow remove a page from Google?

No. Robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it. To keep a page out of search results, use a noindex robots meta tag or an X-Robots-Tag HTTP header, and make sure the page is not blocked in robots.txt so crawlers can see that instruction.

Where do I put the robots.txt file?

Upload it to the root of each host, for example https://www.example.com/robots.txt. Subdomains and different protocols need their own file. It must be named robots.txt in lowercase and saved as UTF-8 plain text.

Do I still need the meta keywords tag?

No. Google does not use the keywords meta tag, and long keyword lists can look spammy. The generator keeps the field optional so you can leave it empty.

Is my data sent anywhere?

No. Everything is generated in your browser. Your settings are saved only on your own device so you can come back to them, and the Reset button clears them.