On some pages the data you want is never in the HTML — the page fetches it from a JSON
endpoint in the background. `capture_network` hands you those responses raw.
Works in `mode: "browser"` only. Capturing costs no extra credit.
Settings
- `capture_network` (boolean, default `false`) — turns capture on.
- `capture_options.resource_types` (string array, default `["xhr","fetch"]`) — which
request types to keep. Images, fonts and scripts are excluded by default.
- `capture_options.url_filter` (string) — a regular expression; only matching URLs are kept.
- `capture_options.max_captured` (int, default `50`, max `200`) — how many responses to keep.
- `capture_options.max_body_bytes` (int, default `10240`, max 256 KB) — per-body limit.
- `capture_options.max_total_bytes` (int, default `512000`, max 2 MB) — total across all bodies.
- `capture_options.include_auth_headers` (bool, default `false`) — while `false`,
`Authorization`, `Cookie`, `Set-Cookie`, `X-CSRF-Token` and `X-API-Key` are replaced with
`<redacted>`.
Rules
Breaking any of these returns `400`:
- `capture_network: true` requires browser mode.
- `capture_options` requires `capture_network: true`.
- `url_filter` must be a valid regular expression.
- Numeric limits must be positive integers. Values above the maximum are not rejected — they
are silently clamped.
There is no separate capture-duration parameter. The capture window is the page window,
controlled by `wait_until`, `wait_seconds` and `actions`. If a late endpoint is being missed,
wait longer.
Full example
curl -X POST http://localhost:3000/webhook/unblocker \
-H "Content-Type: application/json" \
-H "x-api-key: js_YOUR_API_KEY" \
-d '{
"urls": ["https://shop.example.com/product-866178029/reviews"],
"selectors": [
{ "property_name": "page_title", "selector": "title", "attribute_type": "TEXT" }
],
"options": {
"mode": "browser",
"wait_until": "networkidle",
"wait_seconds": 3,
"capture_network": true,
"capture_options": {
"resource_types": ["xhr", "fetch"],
"url_filter": "review",
"max_captured": 50,
"max_body_bytes": 10240,
"max_total_bytes": 512000,
"include_auth_headers": false
}
}
}'Response
Captured traffic comes back next to `Result` in the `check-job` output.
{
"ID": 4412,
"Result": { "page_title": "Product Reviews" },
"network": [
{
"url": "https://apigw.example.com/api/review-read/product-reviews?...",
"method": "GET",
"status": 200,
"resource_type": "xhr",
"content_type": "application/json; charset=utf-8",
"request_headers": { "Cookie": "<redacted>", "Accept": "application/json" },
"response_headers": { "content-type": "application/json" },
"request_body": null,
"response_body": "{\"content\":[{\"userFullName\":\"J** D**\",\"rate\":4,\"comment\":\"...\"}]}",
"started_ms": 5330,
"truncated": false
}
],
"network_dropped": 0
}- `started_ms` — milliseconds since navigation. Use it to tune `wait_seconds`.
- `truncated` — the body hit `max_body_bytes`.
- `network_dropped` — responses discarded because of a limit. Anything above zero means
you should raise the limits or narrow `url_filter`.
A practical workflow
- Run once with no filter and `max_captured: 200` to see which URLs the page calls.
- Once you spot the endpoint, narrow it down with `url_filter`.
- If bodies come back `truncated: true`, raise `max_body_bytes`.
- If the endpoint never appears, it fires late: use `wait_until: "networkidle"` with a longer
`wait_seconds`, or add the `scroll` or `click` action that triggers it.
Leave `include_auth_headers` at `false` unless you have a specific reason. Captured headers
are stored with the job results, and redaction keeps credentials out of them.