🌐 Asynchronous Deep Crawling • Up to 50,000 Pages

Scalable Web Crawler API for High-Volume Data Ingestion

Crawl entire documentation trees, knowledge bases, and multi-hop link graphs with intelligent frontier queues, automatic sitemap traversal, and real-time SSE progress streams.

Start Crawling Free → View Crawl API Specs
POST /v1/crawl 
# Start Recursive Crawl Job
curl -X POST https://api.flycrawl.net/api/v1/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "depth": 2,
    "limit": 50,
    "workers": 16
  }'
🗺️

Automated Sitemap Discovery

Input a root URL and FlyCrawl discovers `sitemap.xml`, `wp-sitemap.xml`, and sub-sitemaps automatically to enqueue all target pages.

📡

Real-Time SSE Streaming

Listen to live Server-Sent Events as pages are scraped, parsed, and tokenized without waiting for the full batch job to conclude.

🛡️

Polite Rate Limiting

Built-in robots.txt compliance, domain concurrency caps, and jitter backoff ensure target servers are respected while maintaining maximum throughput.