Common Crawl Finds llms.txt Files Misused as Crawler Directives

VERA Brief

AI-generated. Grounded in the article and its cited sources.

Common Crawl's analysis of 584,107 llms.txt files found widespread misuse. Many files are templated, lack functional links, and attempt to dictate crawler behavior beyond the format's capabilities, suggesting a misunderstanding of the file's purpose.

Key facts

• Common Crawl analyzed 584,107 llms.txt files.
• A significant number of websites are using llms.txt files like robots.txt.
• Many llms.txt files originated from templates and lacked functional links.
• Some files contained crawler rules the llms.txt format cannot enforce.
• This misuse suggests a misunderstanding of the llms.txt file's intended purpose.