وب اسکرپینگ با مقیاس کلان برای ماشین لرنینگ (ML) و مدل‌های زبانی

1. Stealth Crawling

Bypass Cloudflare Turnstile, DataDome, and anti-scraping WAFs using distributed residential proxies and headless Chromium clusters.

2. Fit-Markdown Cleaning

Heuristically strip navigation bars, advertisements, cookie popups, and SVG glyphs. Retain pure editorial text, markdown tables, and headers.

3. Semantic Chunking

Split documents along semantic boundaries (H2, H3, paragraphs) preserving context rather than blind character token slicing.

4. Export & Embedding

Export directly to JSONL, Parquet or compute native 1536-dim / 3072-dim embeddings ready for direct upsert into Pinecone and Qdrant.

from flycrawl import FlyCrawlApp

app = FlyCrawlApp(api_key="YOUR_FLYCRAWL_API_KEY")

# 1. Scrape & generate clean token-minimized ML training data
result = app.scrape_url(
    "https://en.wikipedia.org/wiki/Deep_learning",
    params={
        "formats": ["markdown", "chunks", "embeddings"],
        "only_main_content": True,
        "embeddings": {
            "dimensions": 3072,
            "model": "native-large"
        }
    }
)

# 2. Output ready for fine-tuning or vector databases
print(f"Token reduction: {result.get('token_saved_pct')}%")
print(f"Clean Markdown: {result['markdown'][:250]}...")
print(f"Vector Dimensions: {len(result['embeddings'][0])}")

Why is Fit-Markdown better than raw HTML for Machine Learning training?

Raw HTML contains over 70% to 90% boilerplate (CSS classes, script tags, tracking pixels, nested navbars). Feeding raw HTML to tokenizers wastes token context windows and teaches models irrelevant web markup instead of semantic facts. Fit-Markdown isolates the core content, slashing training token costs.

Can FlyCrawl scrape dynamic Single-Page Applications (React, Vue, Next.js) for ML datasets?

Yes. FlyCrawl uses a headless Chromium browser cluster with automated DOM wait conditions, JavaScript execution, and infinite-scroll handling to render single-page applications before extracting structured markdown.

What export formats are supported for ML datasets?

FlyCrawl supports Fit-Markdown, Structured JSON, JSONL (JSON Lines) for bulk pretraining corpora, Parquet files, and pre-computed 1536-dimensional and 3072-dimensional vector embeddings.