Tell search engines what to crawl — valid robots.txt with rules and sitemap.
robots.txt sits at your domain root and instructs crawlers which paths they may fetch. The generator assembles the file from your rules — one User-agent block, any number of Allow/Disallow lines, plus your sitemap.
Rules are prefix matches: Disallow: /private/ blocks everything under /private/, while Allow: can carve exceptions back out.
At the domain root: example.com/robots.txt. Anywhere else is ignored.
No! It only asks polite crawlers to stay out. For real privacy, use passwords or noindex.
Not required, but it helps crawlers discover your sitemap faster.
A robots.txt file tells search engines which parts of your site they may crawl. Generate a valid one in seconds — allow everything, block admin areas and staging sites, or point crawlers to your sitemap.
XYZREG makes it easy to search, register and transfer domain names at transparent yearly prices, with .com domains from only $11.99/yr. Manage DNS, protect your WHOIS privacy, and use free online tools including WHOIS lookup, DNS lookup, IP lookup and SSL checker - no account required.