Home / Use Cases / RAG & AI Agents
LLM RETRIEVAL-AUGMENTED GENERATION

Turn Any Website Into
Clean Tokens for RAG & Agents

HTML is noisy, verbose, and wastes precious LLM context windows. FlyCrawl extracts clean Fit-Markdown preserving technical tables, structured lists, and metadata, giving your LangChain, LlamaIndex, and AutoGen pipelines the purest knowledge possible.

✂️

90% Less Token Consumption

Strips boilerplate navigation, inline CSS/JS, and layout tags. Only the pure semantic content reaches your embedding models.

📑

Deterministic YAML Frontmatter

Every response contains Title, Source URL, Description, and Timestamp ready for direct chunk metadata indexing.

📊

Preserved Technical Tables

HTML tables (specs, dimensions, pricing) are accurately converted to standard Markdown tables so LLMs reason accurately.

Python LangChain & OpenAI Example 

import requests
from langchain_text_splitters import MarkdownHeaderTextSplitter

# 1. Fetch clean markdown from FlyCrawl
response = requests.post(
    "https://api.flycrawl.net/api/v1/scrape",
    headers={"Authorization": "Bearer fly_live_your_key"},
    json={"url": "https://example.com", "format": "markdown"}
)
markdown_content = response.json()["data"]["markdown"]

# 2. Split into vector chunks by headers
headers_to_split = [("#", "H1"), ("##", "H2"), ("###", "H3")]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split)
chunks = splitter.split_text(markdown_content)

print(f"Generated {len(chunks)} clean chunks ready for Vector DB!")
Start Building RAG with FlyCrawl ✨