Mastering FlyCrawl: Options, RAG & Vectors
Learn how each parameter powers high-speed AI extraction, how 1536-dim and 3072-dim embeddings transform raw web pages into vector databases, and how transparent credit pricing works.
🚀 5 Steps from Any Web Page to High-Dim Vectors
Click on each step below to inspect the interactive visual walkthrough and mockups.
Enter Target URL or Choose from 200+ Benchmarks
In FlyCrawl, you are never restricted. You can paste any modern single-page application (React, Vue, Next.js), documentation portal, e-commerce catalog, or news website. For fast guest evaluation without logging in, click any of the 200 curated benchmark chips (Wikipedia, GitHub, Stripe, BBC, etc.).
Configure Crawl Depth, Limits & Proxy Geotargeting
Customize how the crawler behaves depending on your goal: whether you want a single fast page or an entire documentation site. Select between Fit-Markdown (clean tokens for LLMs) or JSON (structured format with metadata).
- Depth 0: Single page extract (Fastest, consumes 1 credit).
- Depth 1-3: Recursive network crawler following all internal links.
- Country Proxy: Rotates residential IPs from US, Germany, UK, France, and Japan to defeat Cloudflare/DataDome blocks.
Enable RAG Chunking & Choose Your Vector Model
Toggle the Smart Chunking & Vector switch to convert extracted text into ready-to-search mathematical vectors. Our intelligent chunker splits by header tags (#, ##, ###) while maintaining context overlap.
- ⚡ FlyCrawl Native (1536-dim): Zero cost, lightning 10ms in-memory cosine vectorizer.
- 🔥 FlyCrawl Native Large (3072-dim): Ultra high-dimensional semantic resolution (Free).
- 🟢 OpenAI text-embedding-3-small (1536-dim): Industry standard OpenAI embedding.
- 🚀 OpenAI text-embedding-3-large (3072-dim): Highest MTEB benchmark precision in the world.
Execute Scrape & Watch Live Transparent Credit Calculations
Click Run Scrape. FlyCrawl's dynamic pricing bar calculates exact costs upfront with zero surprises: 1 credit per base page + additional credits for vector embeddings. If the crawl fails or gets blocked by destination, exactly 0 credits are deducted.
1-Click Export to Pinecone, Qdrant & JSONL
Once the crawl finishes, click 'Copy Vector Payload'. The clipboard is instantly loaded with pre-formatted JSON vectors (values array, chunk metadata, source URL, and header section) perfectly formatted for Pinecone, Qdrant, ChromaDB, or pgvector. You can also export the full dataset as JSONL.
{
"upsert_request": {
"namespace": "flycrawl-production",
"model": "text-embedding-3-large",
"dimensions": 3072,
"vectors": [
{
"id": "vec_1725368400_1",
"values": [0.0142, -0.0381, ... 3072 floats],
"metadata": {
"source": "https://en.wikipedia.org/wiki/AI",
"section": "Machine Learning",
"engine": "FlyCrawl High-Dim Engine"
}
}
]
}
}
Every Playground & API Option Explained
Detailed breakdown of what each toggle does, why it matters, and how it impacts credit consumption.
Crawl Depth (Recursive Link Crawler)
Depth 0 scrapes strictly the given URL. Depth 1 parses the page, discovers all internal hyperlinks belonging to the same domain, and extracts them up to your page limit. Depths 2 and 3 continue the tree traversal.
Page Limit & Concurrency Budget
Caps the maximum number of pages extracted during a single crawl session. Executed with 4 concurrent high-performance background workers to ensure lightning throughput without tripping server rate limits.
Country Geotargeting & Anti-Bot Bypass
Routes the crawl through enterprise residential proxies in the selected country. Crucial for websites with geographic paywalls, regional currencies, or strict Cloudflare/DataDome challenges.
Fit-Markdown Engine vs JSON
Eliminates 95% of HTML junk, inline styles, trackers, and navigation clutter. Delivers semantic headings, tables, clean links, and lists that consume up to 80% fewer tokens in Claude and GPT-4.
Smart Semantic Header Chunking
Instead of dumb character slicing, FlyCrawl understands markdown headings (#, ##, ###) and partitions paragraphs naturally. The sliding overlap ensures no semantic boundary is cut in half.
Vector Models (1536-dim & 3072-dim Large)
Choose between FlyCrawl Native Fast (1536-dim, zero cost), Native Large (3072-dim), or official OpenAI text-embedding-3 models. Outputs normalized float vectors mathematically ready for Cosine Similarity.
Scrape with Vector Embeddings in 3 Lines of Code
Compatible with Python, LangChain, LlamaIndex, Cursor, and Windsurf.
import requests
response = requests.post(
"https://flycrawl.net/api/v1/scrape",
headers={"Authorization": "Bearer fly_live_your_api_key"},
json={
"url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
"formats": ["markdown", "embeddings"],
"embeddings": {
"model": "text-embedding-3-large",
"dimensions": 3072
}
}
)
data = response.json()
print(f"Extracted {len(data['chunks'])} chunks with 3072-dim vectors!")
Ready to Test It Live?
Open the interactive Playground, test any website or benchmark, and see the clean Fit-Markdown and high-dim vectors generated in real-time.