Step-by-Step Guide to Configuring Robots.txt AI Crawler Whitelist
Robots.txt AI Crawler Whitelist Configuration Guide
1. Background Overview
With the rise of AI-powered search, leading large language model providers have launched their own web crawlers:
By default, these crawlers follow the rules in robots.txt. To allow your content to be indexed by large AI models, you must explicitly permit access for these crawlers in your robots.txt file.
II. Standard Configuration Template
Allow all AI crawlers
User-agent: GPTBot
Disallow:
User-agent: ClaudeBot
Disallow:
User-agent: Google-Extended
Disallow:
User-agent: Bytespider
Disallow:
User-agent: CCBot
Disallow:
```
Allow AI crawlers but restrict sensitive directories
User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /api/internal/
User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /private/
```
3. Step-by-Step Configuration Process
Step 1: Locate the robots.txt file
Step 2: Edit file content
Open robots.txt in a text editor and add AI crawler rules.
**Attention**:
Step 3: Upload File
Upload the edited robots.txt to your website's root directory.
Step 4: Verify configuration
Visit https://www.example.com/robots.txt，确认修改生效。 in your browser
IV. Advanced Configuration Tips
Allow specific AI crawlers only
Allow OpenAI and Google
User-agent: GPTBot
Disallow:
User-agent: Google-Extended
Disallow:
# Block Other AI Crawlers
User-agent: *
Disallow: /
```
Submit with Sitemap
User-agent: GPTBot
Disallow:
Sitemap: https://www.example.com/sitemap.xml
```
V. Important Notes
1. **Do not completely block AI crawlers**: `Disallow: /` prevents AI from crawling.
2. **Distinguish AI Crawlers from Traditional Crawlers**: Configure separate rules for each.
3. **Periodically review logs**: Monitor actual crawling behavior of AI bots
4. **Balance openness with privacy**: Sensitive data must remain protected
6. Verify AI Crawler Access
Use the following tools to verify that the configuration is active:
