An MCP web scraping server your agent can actually drive

What MCP changes about scraping

A conventional scraper is a program you write. You inspect a page, choose selectors, and re-inspect every time the markup shifts. An MCP scraping server inverts that: it exposes intent-shaped tools, and the model decides which one to call next.

• Tools, not selectors. The agent asks to observe a page or extract prices; it does not need a CSS path that breaks next Tuesday.
• Recovery is in the loop. Every navigation reports the challenge it hit, so the model can escalate from a fast engine to a full browser instead of failing silently.
• Structured output is the default. Results come back as typed JSON, ready for the next tool call rather than for a parser you maintain.

Two engines, one platform

Most scraping products pick a side: a pipe for agents, or a crawler for volume. deepscrape runs both behind one API and one credit balance.

webrain — the agent engine

webrain is the MCP-native engine that powers AI chat and autonomous agents. It is open source (MIT) and local-first, shipped as a single Rust binary of roughly 22 MB with no Node, Python or Docker runtime. It is the same engine available at docs.webrayn.dev — you can install it yourself, or let deepscrape run it for you.

• 17 intent-based tools behind one MCP surface, over stdio or HTTP.
• Three browser engines plus a no-browser path, all behind one CDP backend.
• Offline reading. Video transcripts run on a bundled whisper build and vision on a local Qwen3-VL-2B model, so no cloud key is required.

Crawl4AI — the crawling engine

Crawl4AI is the Python crawler package that handles breadth: BFS, DFS and BestFirst traversal, crash recovery and a prefetch mode for large jobs. It drives the hosted playground and the CrawlPack machines you can deploy into your own account.

What your agent can call

Each tool maps to an intent rather than a browser primitive. A representative slice of the surface:

Extraction without an LLM in the loop

The expensive way to extract data is to paste a page into a model and ask for JSON. The deterministic way is to let the engine probe the DOM and generate the schema itself. That second path is what autoschema does, and it costs nothing per row.

Measured on a live storefront, the agent-driven route returned 407 unique products in about 17 tool calls for roughly 46,000 tokens. Pointing the deterministic extractor at the same store produced the same 407 products with zero tokens and no model in the loop. Which one you want depends on whether the page needs judgement or just parsing.

Four ways to load a page

Sending a full browser at a static HTML page is the most common way to waste money. The engine can be selected per operation:

The runtime every run inherits

The engine is the visible half. The half that decides whether a job survives contact with the open web is the runtime underneath it:

• A memory-adaptive dispatcher. Work is scheduled against a pre-warmed browser pool, so runs do not pay a cold-start penalty per page.
• Persistent stealth profiles. Real browser profiles that survive between runs, rather than a fresh fingerprint every request.
• An encrypted credential vault. Logins are stored with AES-256-GCM, with optional TOTP, and cookies transfer between engines.
• Proxy rotation with real geography. Egress across 40+ countries so you can see the regional version of a page rather than a generic one.
• Challenge handling as a first-class signal. Cloudflare, Turnstile and reCAPTCHA gates are reported and cleared, not swallowed.

Run it yourself, or run it managed

Being straight about this matters. webrain is MIT-licensed and local-first: you can install the binary, point Claude, Cursor, Copilot, Codex or Gemini at it over MCP, and pay nothing. If that is all you need, take it.

deepscrape is for the part that starts after that: you would rather not run a browser fleet, rotate your own proxies, keep profiles patched, or own the uptime. You get the same engines as a managed API, with cloud browsers, residential egress and stealth profiles included, billed in credits. Free includes 100 credits per month; Pro is €19.99 for 5,000. If you want the crawler in your own infrastructure instead, CrawlPack machines deploy into your account.

Frequently asked

What is an MCP web scraping server?

It is a server that exposes scraping and browser automation as MCP tools, so an LLM client can drive a browser directly. Instead of writing selectors, you describe the outcome and the model selects tools such as navigate, observe, interact and extract.

Which MCP clients does it work with?

Any MCP client. That includes Claude, Claude Code, Cursor, GitHub Copilot, Codex and Gemini. It speaks stdio and HTTP, so registration is a couple of lines in the client's MCP config.

Do I need to run Node, Python or Docker?

Not for the agent engine. webrain is one static Rust binary; the browsers it drives download on demand into a cache directory. The crawling engine is a Python package, which is one reason people hand that half to us instead.

Can it scrape pages that require a login?

Yes. Credentials live in an encrypted vault with optional TOTP, and a persistent browser profile carries the session across requests and between runs. That covers the common case of a page that only renders once you are authenticated.

Does every extraction need an LLM?

No, and it usually should not. Deterministic modes including autoschema, schema, regex, table, JSON-LD and BM25 run with no model and no token cost. Reserve the agent loop for pages that genuinely need judgement.

Is webrain the same thing as deepscrape?

No. webrain is the open-source engine, available at docs.webrayn.dev and on GitHub. deepscrape is the managed platform that runs it alongside the Crawl4AI crawler, with hosting, proxies, profiles and billing handled for you.

Where to next

• The deep scraper — what "deep" means when a page renders client-side and fights back.
• deepscrape vs self-hosted Crawl4AI — the honest comparison if you are already running the Python package.
• Architecture and pricing — the full data flow, the two engines and the credit model in one place.
• Talk to us — for volume, procurement or a workload you are not sure we handle.