robots

3 related documents.

Cloudflare's Default AI Crawler Block, September 15, 2026: Will Your Site Vanish from AI Answers?Sep 15, 2026

A breakdown of the Search, Agent, and Training defaults Cloudflare put in force on September 15, 2026. Search crawlers stay allowed, while Agent blocks and mixed-purpose crawler inheritance are the real paths that cut citations in AI answers.

Korean Blog Platform AI Crawler Policies Compared 2026: Naver Blog, Tistory, and Brunch robots.txtSep 11, 2026

Naver Blog blocks GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot outright in robots.txt. Tistory and velog carry no AI-crawler-specific rules, and Brunch blocks training bots only. Here is what each platform's raw robots.txt says and how that changes an article's odds of being cited in AI search.

Managing AI Crawlers: Allowing GPTBot, ClaudeBot, and PerplexityBot and the Trade-offsJun 6, 2026

How to identify GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended, and the visibility trade-offs of allowing or blocking them in robots.txt: based on OpenAI's and Google's official documentation.