DEV3 Technologies home

Developer tool

Robots.txt and Sitemap Generator

Build a robots.txt with a visual rules builder and presets, then paste your URLs to get a valid sitemap.xml. Download both files, all in your browser.

Free, no sign-up. Runs in your browser; nothing is uploaded.

Used for the Sitemap line and to complete relative URLs.

Build

Start from a preset

Rule group 1

* means every crawler. Or a bot name such as Googlebot.

Sitemaps

robots.txt

User-agent: *
Disallow: /admin
Disallow: /api/
Disallow: /login
Disallow: /register
Disallow: /dashboard

Upload it to your site root so it loads at https://example.com/robots.txt.

Checks

  • robots.txt is a request, not security. Do not rely on it to hide private pages, and note a blocked URL can still be indexed if other sites link to it.

How to use the robots.txt and sitemap generator

Two small files that guide crawlers

A robots.txt file tells crawlers which parts of your site to skip. A sitemap.xml lists the pages you want found. Together they help search engines spend their time on the pages that matter.

Build robots.txt

  1. Enter your website address so the Sitemap line is filled in correctly.
  2. Choose a preset close to your setup: allow everything, block everything for a staging site, WordPress, a web app, an online store or a rule set for content-scraping bots.
  3. Edit the groups. Each group has a User-agent (* means all crawlers) followed by Disallow and Allow paths, one per line. A more specific Allow can re-open a path inside a blocked folder.
  4. Read the Checks panel. It warns about paths that do not start with a slash and about the classic mistake of leaving Disallow: / on a live site.

Build sitemap.xml

Switch to the sitemap view and paste URLs one per line. Full URLs work as they are, and paths such as /pricing are completed with your website address. The tool removes duplicates and fragments, skips lines that are not valid URLs and escapes characters like & so the XML stays valid.

Publish and verify

  • Upload robots.txt and sitemap.xml to your site root.
  • Open both URLs in a browser to confirm they load as plain text or XML.
  • Submit the sitemap in Google Search Console and check for errors after a few days.

What to block and what not to

Block internal search results, carts, login pages and admin areas. Do not block CSS or JavaScript files that pages need to render, because search engines use them to understand the layout. Never put secrets in robots.txt, since anyone can read it.

If your site has more than a few hundred pages, generate the sitemap from your CMS or framework automatically and use this tool for checking or small sites.

Frequently asked questions

Where do I put robots.txt?

In the root of your domain, so it loads at https://example.com/robots.txt. Each subdomain needs its own file. Files in subfolders are ignored by crawlers.

Does Disallow remove a page from Google?

Not reliably. It stops crawling, but a blocked URL can still appear in results if other sites link to it. To keep a page out of search, allow crawling and add a noindex tag, or protect it with a login.

Do I need changefreq and priority in my sitemap?

No. Google ignores both values. A clean list of canonical URLs, plus an accurate lastmod where you can provide one, is what matters.

How many URLs can one sitemap hold?

Up to 50,000 URLs and 50 MB uncompressed. Larger sites split the list into several files and reference them from a sitemap index.

How do I submit the sitemap?

Add the Sitemap line to robots.txt, which this tool does for you, and also submit the URL in Google Search Console and Bing Webmaster Tools to see coverage reports.