AI Search Insights

Original guides and experiments on how AI systems crawl, read and cite websites. Every article links to a free tool you can act with.

Crawling & Access · Jul 29, 2026

Robots.txt ambiguities for AI crawlers: blocked or not?

We ran 18 ambiguous robots.txt cases through a real parser and tested them against AI bot tokens. Six of them behave differently depending on who reads them.

Read the article →