Robots.txt & Meta Tag Generator
Choose which pages crawlers can visit and how search engines should treat your content. Clean, standards-aware code appears instantly as you type β no coding needed.
- β‘ Live code preview
- π§ͺ URL tester
- β SEO checklist
- π Open Graph & X cards
- β¬ Download files
Instant live preview
Every change updates your robots.txt and meta tags immediately, with syntax highlighting.
Smart SEO checklist
Green, yellow and red checks explain conflicts, risky rules and missing tags in plain words.
Test any URL
Check whether a page or file would be allowed or blocked for Googlebot, Bingbot or any crawler.
Multiple crawler groups
Set different rules for Googlebot, Bingbot or custom bots, or block common AI crawlers in one click.
Social & search tags
Open Graph and X Card tags with search and share previews, reusing your title and description.
Private by default
Everything is generated in your browser. Nothing you type is uploaded.
Robots.txt and meta tag generator
1 Website & sitemap
2 Who can crawl what?
Only needed when you do not want these files crawled.
3 Extra crawler groups
Give specific crawlers their own rules. Everything else follows the default group above.
βοΈ Advanced directives
Only Yandex ever used this, and it is deprecated. Google and Bing ignore it.
Placed after your groups. Include a User-agent line before any custom Allow or Disallow rules.
π§ͺ Test a URL against these rules
π Basic SEO tags
π¦ Robots meta directives
π£ Open Graph tags (Facebook, LinkedIn)
π X (Twitter) Card tags
π Search & share previews
SEO validation
Checking your configurationβ¦
From generated code to a live website in four steps
Upload robots.txt
Save the file as
robots.txtand upload it to your site root so it opens athttps://yourdomain.com/robots.txt.Test it
Open the robots.txt report in Google Search Console and check that key pages are not blocked. Use the URL tester above first.
Add the meta tags
Paste the meta tags inside the
<head>of each page. Use a different title, description and canonical URL for every page.Recheck after changes
Crawlers cache robots.txt, so changes can take a while to apply. Review the file whenever you add sections or move pages.
Robots.txt does not guarantee deindexing
Robots.txt controls crawler access, not whether a URL is indexed. A blocked page can still appear in results if other sites link to it. To remove a page from search, add a noindex robots meta tag in the page HTML, or send an X-Robots-Tag HTTP header for files such as PDFs. A page blocked in robots.txt may never be crawled, so crawlers cannot discover its noindex instruction. Also remember that robots.txt is public: never list paths you want to keep secret.
Questions, answered
What is a robots.txt file?
A robots.txt file is a plain text file at the root of your website that tells crawlers which URLs they may or may not request. It is a set of polite instructions that reputable search engines follow, not a security feature.
Does Disallow remove a page from Google?
No. Robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it. To keep a page out of search results, use a noindex robots meta tag or an X-Robots-Tag HTTP header, and make sure the page is not blocked in robots.txt so crawlers can see that instruction.
Where do I put the robots.txt file?
Upload it to the root of each host, for example https://www.example.com/robots.txt. Subdomains and different protocols need their own file. It must be named robots.txt in lowercase and saved as UTF-8 plain text.
Do I still need the meta keywords tag?
No. Google does not use the keywords meta tag, and long keyword lists can look spammy. The generator keeps the field optional so you can leave it empty.
Is my data sent anywhere?
No. Everything is generated in your browser. Your settings are saved only on your own device so you can come back to them, and the Reset button clears them.