Crawl API
Whole sites,
streamed as they land.
Point it at a seed URL and receive every page back as clean markdown over a live SSE stream — no polling, no waiting for the whole job.
pricing
$0.001per page
Pay-as-you-go. No subscription, no minimum — and your credits never expire.
SSE
live stream
1–3+
depth control
robots
respected
$0.001
per page, only
Real-time SSE
Results stream as each page completes — start processing immediately.
Depth control
Tune how far the crawler follows links, from shallow to deep.
Advanced proxy
Bot detection handled; robots.txt respected by default.
Control
As shallow — or as deep — as you need.
max_depth = 1
Only pages directly linked from the seed URL. Fast and focused.
max_depth = 2
The seed plus everything those pages link to. Ideal for documentation.
max_depth = 3+
Deep crawling for large surfaces. Use with care on big sites.
what arrives on the stream
pageClean content for each crawled page, the moment it completesusageCost summary once the crawl finishesdoneCompletion signal with total response timeIn practice
One call. That's the integration.
01from llmlayer import LLMLayerClient0203client = LLMLayerClient(api_key="YOUR_LLMLAYER_API_KEY")0405for frame in client.crawl(06 url="https://docs.example.com",07 max_pages=20,08 max_depth=2,09):10 if frame.type == "page":11 print(frame.url, len(frame.markdown)) # pages stream as they land
Start with nothing to lose.
Free credits on signup, no card. The meter only moves when you call.