← WebExtract · API docs

WebExtract for LangChain

langchain-webextract on PyPI (Python) and @webextract/langchain on npm (LangChain.js). The tools are named extract_url and page_markdown, the same as the MCP tools, and return the API JSON as a string.

Python

pip install langchain-webextract

Depends on webextract-sdk (the Python client, imported as webextract; httpx) and langchain-core. Python ≥ 3.10.

from langchain_webextract import WebExtractExtractUrl, WebExtractPageMarkdown, get_webextract_tools
from webextract import WebExtract

tools = get_webextract_tools()                                    # free tier, or WEBEXTRACT_API_KEY if set
tools = get_webextract_tools(api_key="lx_...")                     # prepaid credits (SecretStr field)
tools = get_webextract_tools(client=WebExtract(api_key="lx_..."))  # or pass a client
print(WebExtractPageMarkdown().invoke({"url": "https://example.com"}))

# give them to any tool-calling chat model
model_with_tools = chat_model.bind_tools(tools)

Billing

WEBEXTRACT_API_KEY: used when no api_key is given. An explicit key wins. The variable is ignored when a payment_client is passed or billing is not "key".

x402 (async and sync)

import os
from eth_account import Account
from x402 import x402Client, x402ClientSync
from x402.http.clients import x402HttpxClient, x402_requests
from x402.mechanisms.evm.exact import ExactEvmScheme
from webextract import AsyncWebExtract, WebExtract
from langchain_webextract import WebExtractExtractUrl

signer = Account.from_key(os.environ["X402_PRIVATE_KEY"])   # your key, your process

# async: tool.ainvoke(...)
ax = x402Client(); ax.register("eip155:8453", ExactEvmScheme(signer=signer))
tool = WebExtractExtractUrl(async_client=AsyncWebExtract(payment_client=x402HttpxClient(ax, timeout=60)))
await tool.ainvoke({"url": "https://web-scraping.dev/product/1"})

# sync: tool.invoke(...)
sx = x402ClientSync(); sx.register("eip155:8453", ExactEvmScheme(signer=signer))
tool = WebExtractExtractUrl(client=WebExtract(payment_client=x402_requests(sx)))
tool.invoke({"url": "https://web-scraping.dev/product/1"})

A 402 settlement_failed (verified, not settled, not charged) is retried once with a freshly signed payment (settlement_retries, default 1).

Sandbox rate limit: 20 requests per minute per IP (HTTP 429 with Retry-After). Every x402 payment uses 2 requests (the unpaid 402, then the signed retry), so that is about 10 paid calls per minute.

Errors

By default API errors come back to the model as a JSON string, so an agent can react to free_limit_reached, robots_disallowed and so on:

{"ok": false, "error": {"error": "payment_required", "status": 402, "message": "...", "charged": false,
  "retryable": false, "reason": "free_limit_reached", "hint": "...", "networks": [...],
  "buy_url": "https://webextract.dev/#pricing", "free_remaining": 0}}

Set on_error="raise" to raise webextract.WebExtractError subclasses instead:

The package passes LangChain's standard tool tests (langchain-tests ToolsUnitTests and ToolsIntegrationTests).

LangChain.js

npm i @webextract/langchain @langchain/core zod

import { webextractTools, extractUrlTool, pageMarkdownTool } from "@webextract/langchain";

const tools = webextractTools();                       // free tier, or WEBEXTRACT_API_KEY if set
const tools2 = webextractTools({ apiKey: "lx_..." });  // prepaid credits
console.log(await extractUrlTool().invoke({ url: "https://web-scraping.dev/product/1" }));

Options are the same as the @webextract/sdk JS client: apiKey, paymentFetch (x402 via @x402/fetch, see the AI SDK page), billing ("free", "key", "x402", "x402-testnet"), baseUrl, timeoutMs. Errors come back as {"ok":false,"error":{...}}; set onError: "throw" to throw.

Response fields

Read product.price_detail for prices (never 0; status: "unresolved" comes with unresolved.reason), content_source for where the content came from, and product_unresolved when product is null. Full schema: API docs · /openapi.json.

Also on this site: Vercel AI SDK · MCP.