CoWriter AI - AI Crawler Configuration

ABOUT
CoWriter AI is an expert AI-driven writing assistant for academic and professional writing.

ORGANIZATION
Name: CoWriter AI
Website: https://cowriter.ai
Email: support@cowriter.ai
Privacy Policy: https://cowriter.ai/privacy

ALLOWED CONTENT
- Homepage and marketing pages (/)
- Blog posts (/blog/*)
- Documentation (/docs/*)
- Pricing (/pricing)
- FAQ and support content

RESTRICTED CONTENT
- API endpoints (/api/*)
- Admin interfaces (/admin/*)
- Checkout pages (/checkout/*)
- User accounts and private content

SUPPORTED CRAWLERS
OpenAI: GPTBot, ChatGPT-User - Full access to public content
Anthropic: ClaudeBot - Full access to public content
Google: Google-Extended - Full access to public content
Common Crawl: CCBot - Full access to public content
Perplexity: PerplexityBot - Full access to public content
You.com: YouBot - Full access to public content
Omgili: OmgiliBot - Full access to public content

CRAWLING GUIDELINES
- Crawl delay: 5 seconds
- Max concurrent connections: 2
- Respect HTTP status codes and headers
- Peak hours (09:00-17:00 UTC): reduce crawl rate

RESOURCES
Sitemap: https://cowriter.ai/sitemap.xml
JSON Config: https://cowriter.ai/llms.json
Robots.txt: https://cowriter.ai/robots.txt

TERMS
Permitted: AI training, indexing, summarization
Restrictions: Respect access controls, comply with data protection laws
Commercial reuse subject to Terms of Service

CONTACT
Email: support@cowriter.ai
Subject: AI Crawler - [Your Issue]
Response time: 48 hours

Last updated: 2025-11-02