AI.txt Checker
Check if a website provides AI crawler rules and analyze its AI.txt file for completeness and best practices.
What is AI.txt?
AI.txt is a proposed standard that helps website owners communicate their AI usage policies to AI crawlers, large language models, and other AI systems. Similar to how robots.txt tells search engine crawlers what they can access, AI.txt specifically addresses AI-related activities like training, summarization, and indexing. By placing an ai.txt file at the root of your domain, you make your content policies machine-readable for AI agents.
Why Should Websites Create AI.txt?
Control AI Usage
Specify whether AI systems can crawl, train on, or summarize your content. Set clear boundaries for how your data is used.
Agent Management
Allow or block specific AI agents like OpenAI, Anthropic, or Google Gemini from accessing your content.
Future-Proofing
As more AI crawlers emerge, having an AI.txt ensures your policies are discoverable without updating multiple systems.
Complement robots.txt
While robots.txt handles search crawlers, AI.txt specifically addresses AI-related use cases that robots.txt wasn't designed for.
How Do AI Crawlers Use AI.txt?
AI crawlers and language models check for AI.txt at /ai.txt or /.well-known/ai.txt when visiting a domain. The file tells them what types of AI activities are permitted and which agents are allowed or blocked. This is increasingly important as AI companies develop responsible crawling practices.
AI.txt Example
Here is the AI.txt file used by AI Website Detector. You can use it as a reference for your own:
site: aiwebsitedetector.com name: AI Website Detector description: AI Website Detector detects AI website builders, CMS platforms, frameworks, and hosting providers from any URL. ai_crawling: allowed ai_training: allowed ai_summarization: allowed ai_indexing: allowed allowed_agents: openai anthropic google-gemini perplexity contact: https://aiwebsitedetector.com/contact sitemap: https://aiwebsitedetector.com/sitemap.xml
Also check your LLM.txt
LLM.txt is a complementary standard focused on providing context to large language models. Use our LLM.txt Checker to analyze your LLM.txt file.