robots.txt

robots.txt is a web-based platform providing free SEO tools and resources for managing and optimizing robots.txt files. The service offers utilities such as a robots.txt tester, HTTP status checker, and comprehensive guides on AI user-agents to assist developers and SEO professionals in controlling search engine crawler access.

LLM mention score

Normalized 0–100 · last 8 weeks

DataForSEO API

Get LLM mention data of any company via DataForSEO API

Get access to the structured data on keyword, brand, and website mentions in LLMs, including metrics like AI search volume, impressions, and mentions count.

Get Free API Credits API Documentation

How to get LLM mention data →

// Fetch robots.txt mentions
POST v3/ai_optimization/llm_mentions/search/live
[
    {
        "target": [
            {
                "keyword": "robots.txt",
                "search_scope": ["any"]
            }
        ],
        "platform": "chat_gpt",
        "order_by" : ["ai_search_volume,desc"]
    }
]