Technical SEO

Crawlers & AI

Control search crawlers, AI discovery files and dynamic sitemap content.

ROBOTSrobots.txtLive rules ↗SEARCHsitemap.xmlGenerated URLs ↗LLM INDEXllms.txtAI-readable content ↗AI POLICYai.txtUsage policy ↗

robots.txt

Choose which crawlers can access the public website.

Active

Enable robots.txt crawlingMaster switch for crawler access.Allow search enginesGoogle, Bing and other standard search crawlers.Allow AI crawlersGPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot.Disallowed paths One path per line

Dynamic sitemap

Published, indexable content is added automatically.

Active

Enable sitemapExpose the XML sitemap and reference it in robots.txt.

Published postsExclude posts marked noindex.CategoriesDynamic category landing pages.ServicesInclude the services and contact page.NewsletterInclude the newsletter page.

Maximum posts

llms.txt

Structured discovery index for AI assistants and answer engines.

Active

Publish llms.txtReturn the live AI-readable content index.Site introductionUsage and citation instructions

Include postsTitles, canonical URLs and quick answers.Include categoriesTopic names, URLs and descriptions.Include servicesCommercial services and contact URL.

ai.txt policy

Machine-readable permissions and attribution policy.

Active

Publish ai.txtMake the AI policy available publicly.Require attributionAsk AI systems to cite the canonical source URL.Allow model trainingExplicitly permit content use for training. Keep off unless intended.Custom AI usage policy