The most common cause of empty fields is timing: the content had not loaded yet when the
selectors ran. The three settings on this page fix that.
wait_until — the load strategy
- `"commit"` — as soon as response headers arrive. Static HTML where the full content is
in the first response.
- `"domcontentloaded"` — when the HTML has been parsed. The default, and right for
most pages.
- `"load"` — when all resources have loaded. Pages where content appears after assets load.
- `"networkidle"` — after 500 ms with no network requests. Heavy SPAs that fetch data
over XHR.
wait_seconds — a fixed extra wait
Seconds to wait after the load strategy completes; the default is `1`. Useful for content
that animates in or renders on a delay.
wait_selector — wait for the target element
The most reliable option: "wait until this element exists, then read".
{
"options": {
"wait_selector": ".product-grid",
"timeout_seconds": 60
}
}If the selector never appears within the timeout, the request fails.
Settings by scenario
- Static HTML — `wait_until: "commit"`.
- Server-rendered with light JavaScript — `wait_until: "domcontentloaded"` (default).
- SPA that fetches data — `wait_until: "load"`, `wait_seconds: 3`.
- Heavy SPA (React, Angular) — `wait_until: "networkidle"`, `wait_seconds: 5`.
- You know the target element — `wait_selector: ".product-list"`.
- Unpredictable render time — `wait_until: "networkidle"`, `wait_seconds: 3`.
Full example — a heavy SPA
curl -X POST http://localhost:3000/webhook/unblocker \
-H "Content-Type: application/json" \
-H "x-api-key: js_YOUR_API_KEY" \
-d '{
"url": "https://shop.example.com/women/dresses",
"selectors": [
{
"property_name": "products[]",
"selector": ".product-grid-product",
"attribute_type": "OBJECT",
"children": [
{ "property_name": "name", "selector": ".product-name", "attribute_type": "TEXT" },
{ "property_name": "price", "selector": ".price__amount", "attribute_type": "TEXT" }
]
}
],
"mode": "browser",
"options": {
"wait_until": "networkidle",
"wait_seconds": 5,
"timeout_seconds": 120,
"use_proxy": true
}
}'Related settings
- `humanize` (default `true`) — mouse movement and smooth scrolling make bot detection
harder.
- `use_proxy` (default `true`) — leave it on for geo-restricted or rate-limited targets.
- `retry_count` (default `2`) — automatic retries on transient failures.
- `headers` — one `key:value` per line for a custom `User-Agent`, `Accept-Language` and so on.
`wait_until: "networkidle"` with `wait_seconds: 5` easily exceeds 30 seconds in total. Raise
`timeout_seconds` to match, or the request is cut off before it finishes.