SEO Tools

Robots.txt Generator

Generate a valid robots.txt file online for free — allow or disallow paths per user-agent, add a sitemap reference, instantly inside your browser.

Private by default Instant results No signup
Robots.txt GeneratorLive
User-agent groups
Test a URL against these rules

Allowed · Group * · Matched rule: no matching rule (default allow)

robots.txt
User-agent: *
Disallow: /admin/
Disallow: /private/
0 B uploaded · everything runs on this deviceSEO · Tofuwebtools.com
Overview

What is robots.txt?

robots.txt is a plain text file at a site's root that tells search engine crawlers which parts of the site they're allowed or not allowed to visit. It's a request, not an enforcement mechanism — well-behaved crawlers like Googlebot respect it, but it doesn't hide content from being accessed directly by URL.

This generator builds rule groups per user-agent, with Allow and Disallow paths and an optional sitemap reference.

Guide

How to use it

1Add a rule group and set the user-agent (or * for all)
2List paths to disallow, and any to explicitly allow
3Add your sitemap URL
4Copy the result into robots.txt at your site's root

How the file is built

Each group starts with a User-agent: line, followed by any number of Disallow: and Allow: lines, and the file ends with an optional Sitemap: line pointing to your sitemap's full URL.

Why it helps

Features & benefits

Multiple user-agent rule groups
Allow and Disallow paths per group
Optional sitemap reference line
100% private — nothing leaves the device

Why this runs entirely in your browser

Path rules can reveal unlaunched sections of a site, so nothing is uploaded. The file is assembled in JavaScript locally in the tab, which means it keeps working offline once the page has loaded.

Common uses

Blocking crawlers from admin or staging paths, allowing full access while pointing to a sitemap, or setting different rules for specific bots like image or ad crawlers.

Frequently asked questions

A few things people usually want to know before trusting the numbers.

Not reliably — it blocks crawling, but a disallowed URL can still be indexed if linked from elsewhere. Use a noindex meta tag to prevent indexing.

At the root of the domain — for example, https://example.com/robots.txt — it won't be recognized anywhere else.

It applies the rules that follow to every crawler that doesn't have its own more specific rule group.

Yes — add one Sitemap line per sitemap file, or point to a sitemap index file that lists them all.

No. The file is generated locally in your browser. Nothing entered here is sent to a server, stored, or shared.