crawler.sh

crawler.sh is a local-first web crawler and markdown extractor designed for AI training, RAG pipelines, and fine-tuning corpora. It converts websites into clean, structured Markdown by rendering JavaScript SPAs with a custom engine, eliminating the need for headless Chrome or cloud-based APIs. The tool supports polite crawling with robots.txt compliance and includes built-in SEO analysis features for data quality assurance.

LLM mention score

Normalized 0–100 · last 8 weeks

DataForSEO API

Get LLM mention data of any company via DataForSEO API

Get access to the structured data on keyword, brand, and website mentions in LLMs, including metrics like AI search volume, impressions, and mentions count.

Get Free API Credits API Documentation

How to get LLM mention data →

// Fetch crawler.sh mentions
POST v3/ai_optimization/llm_mentions/search/live
[
    {
        "target": [
            {
                "keyword": "crawler.sh",
                "search_scope": ["any"]
            }
        ],
        "platform": "chat_gpt",
        "order_by" : ["ai_search_volume,desc"]
    }
]