Crawl jobs
A job is a batch of URLs. Each URL becomes one independent task, so a slow or failing page never holds up the rest of the batch.
Create a job
Body
| Field | Type | Default | Notes |
|---|---|---|---|
urls | string[] | required | 1–1000 entries, http or https |
resultTtlSeconds | integer | 600 | 60–604800. How long the HTML stays downloadable |
package | string | "regular" | "advance" adds browser handling for more demanding pages and can attempt supported CAPTCHA challenges as part of scraping, at the Advance price. A host configured to require Advance is routed that way; the response lists it in advanceRequiredFor. |
options | object | — | See below |
Render options
| Option | Default | What it does |
|---|---|---|
enableJavaScript | true | Turn off only for static pages; it is much faster |
blockAssets | true | Drops images, video and SVG from the DOM. Cuts size a lot. Set false if image URLs matter to you |
navigationTimeoutMs | 120000 | How long to wait for the page to load |
postLoadDelayMs | 1000 | Pause after load, before reading the DOM |
scrollToBottom | false | Scroll to trigger lazy-loaded content |
scrollTimeoutMs | 45000 | Cap on the scrolling phase |
expectedText | — | Treat the render as failed unless this string appears. The strongest tool you have |
screenshot | false | Also take a picture of the page, downloadable from /v1/tasks/{taskId}/screenshot. Forces blockAssets off, so the request is slower and heavier |
postLoadDelayMs is the option people get wrong.
Pages that navigate or hydrate with JavaScript after load need 3,000–5,000 ms.
With the 1,000 ms default you may capture the page mid-transition and get
Execution context was destroyed instead of content.
Reject the wrong page with expectedText
A site can answer with a page that looks fine to a browser but is not the one
you wanted: a consent wall, an empty search, a "something went wrong" page.
Give expectedText a string that only the right page contains, such
as an attribute every listing carries. A page without it is retried at no
cost, and if it never appears the task ends FAILED with
errorCode: "EXPECTED_TEXT_MISSING".
expectedText, the last attempt is charged like a successful
request: the work was done, and a mistyped string must not make a crawl free.
The task shows "billed": true, and its HTML stays downloadable from
GET /v1/tasks/{taskId}/html so you can see what the page said.
Blocked pages and timeouts are never billed.
- It is matched against the rendered HTML, not the visible text, so markup such as
data-listing-idworks. - The match is exact and case-sensitive; up to 1,000 characters.
- Pick something structural rather than a word in the content: prices and titles change, the attribute does not.
import os
import time
import requests
API = os.environ.get("API", "https://bankworks.info")
POLL_SECONDS = float(os.environ.get("POLL_SECONDS", "5"))
http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['API_KEY']}"
# Only a page that contains this text counts as a success. A consent wall, an
# empty search or a "something went wrong" page is retried; if the text never
# appears the task fails, and that last attempt is billed (its HTML is kept).
created = http.post(f"{API}/v1/jobs", json={
"urls": ["https://example.com/cars?page=1", "https://example.com/cars?page=2"],
"options": {"expectedText": "data-listing-id", "postLoadDelayMs": 3000},
}, timeout=30)
created.raise_for_status()
job_id = created.json()["jobId"]
while True:
body = http.get(f"{API}/v1/jobs/{job_id}", params={"limit": 100}, timeout=30).json()
job = body["job"]
if job["doneTasks"] + job["failedTasks"] >= job["totalTasks"]:
break
time.sleep(POLL_SECONDS)
for task in body["tasks"]:
if task["status"] == "DONE":
print(f"ok: {task['url']}")
elif task.get("errorCode") == "EXPECTED_TEXT_MISSING":
print(f"wrong page: {task['url']}")
else:
print(f"failed: {task['url']} ({task.get('errorCode')})")
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");
const POLL_MS = Number(process.env.POLL_SECONDS ?? 5) * 1000;
const headers = { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json" };
// Only a page that contains this text counts as a success. A consent wall, an
// empty search or a "something went wrong" page is retried; if the text never
// appears the task fails, and that last attempt is billed (its HTML is kept).
const created = await fetch(`${API}/v1/jobs`, {
method: "POST",
headers,
body: JSON.stringify({
urls: ["https://example.com/cars?page=1", "https://example.com/cars?page=2"],
options: { expectedText: "data-listing-id", postLoadDelayMs: 3000 },
}),
});
if (!created.ok) throw new Error(`HTTP ${created.status}: ${await created.text()}`);
const { jobId } = await created.json();
let body;
for (;;) {
body = await (await fetch(`${API}/v1/jobs/${jobId}?limit=100`, { headers })).json();
const { doneTasks, failedTasks, totalTasks } = body.job;
if (doneTasks + failedTasks >= totalTasks) break;
await new Promise((resolve) => setTimeout(resolve, POLL_MS));
}
for (const task of body.tasks) {
if (task.status === "DONE") console.log(`ok: ${task.url}`);
else if (task.errorCode === "EXPECTED_TEXT_MISSING") console.log(`wrong page: ${task.url}`);
else console.log(`failed: ${task.url} (${task.errorCode})`);
}
API="${API:-https://bankworks.info}" # your key is read from $API_KEY
# Only a page that contains this text counts as a success. A consent wall, an
# empty search or a "something went wrong" page is retried; if the text never
# appears the task fails, and that last attempt is billed (its HTML is kept).
job_id=$(curl -sS -X POST "$API/v1/jobs" \
-H "Authorization: Bearer $API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"urls": ["https://example.com/cars?page=1", "https://example.com/cars?page=2"],
"options": { "expectedText": "data-listing-id", "postLoadDelayMs": 3000 }
}' | jq -r '.jobId')
while :; do
state=$(curl -sS -H "Authorization: Bearer $API_KEY" "$API/v1/jobs/$job_id?limit=100")
printf '%s' "$state" | jq -e '.job.doneTasks + .job.failedTasks >= .job.totalTasks' >/dev/null && break
sleep "${POLL_SECONDS:-5}"
done
printf '%s' "$state" | jq -r '.tasks[]
| if .status == "DONE" then "ok: \(.url)"
elif .errorCode == "EXPECTED_TEXT_MISSING" then "wrong page: \(.url)"
else "failed: \(.url) (\(.errorCode))" end'
Example
import os
import requests
API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]
response = requests.post(
f"{API}/v1/jobs",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"urls": ["https://example.com/"],
"resultTtlSeconds": 3600,
"options": {
"enableJavaScript": True,
"postLoadDelayMs": 5000,
},
},
timeout=30,
)
response.raise_for_status()
job = response.json()
print(f"job {job['jobId']} with {job['taskCount']} task(s)")
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");
const response = await fetch(`${API}/v1/jobs`, {
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
urls: ["https://example.com/"],
resultTtlSeconds: 3600,
options: { enableJavaScript: true, postLoadDelayMs: 5000 },
}),
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${await response.text()}`);
}
const job = await response.json();
console.log(`job ${job.jobId} with ${job.taskCount} task(s)`);
API="${API:-https://bankworks.info}" # your key is read from $API_KEY
curl -sS -X POST "$API/v1/jobs" \
-H "Authorization: Bearer $API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"urls": ["https://example.com/"],
"resultTtlSeconds": 3600,
"options": {
"enableJavaScript": true,
"postLoadDelayMs": 5000
}
}'
https://example.com/ is a placeholder. Replace it with a page
you have the right to collect.
Response
{"ok": true, "jobId": "0199…", "taskCount": 20, "deliveryDeferred": 0,
"package": "regular", "packages": {"regular": 20, "advance": 0}}
deliveryDeferred counts tasks whose queue delivery is being
retried from the durable outbox. A positive number is not an error — the
work is committed and will run; it just was not handed to the queue on the
first attempt.
Regular or Advance
Every request is either Regular or Advance.
You choose with the top-level package field, on the same endpoint
and with the same key. Leave it out for Regular.
Both packages render pages in Chrome. Advance is an integrated Web Scraping API
option, not a standalone CAPTCHA-solving service; site restrictions may still
prevent retrieval.
| Regular | Advance | |
|---|---|---|
| Send | nothing, or "package": "regular" | "package": "advance" |
| Browser context | Fresh, isolated context for the request | Worker browser context; site cookies may persist between requests |
| More demanding pages | Standard browser rendering | Additional browser handling, including attempts to handle supported CAPTCHA challenges encountered during scraping |
| Unsupported or unsolved challenge | The task fails with CAPTCHA_REQUIRED | The task may fail with CAPTCHA_UNSOLVED |
| Price | The standard price per 1,000 | The Advance price per 1,000 (see pricing) |
| Not billed | Blocked pages and timeouts | Blocked pages, timeouts, and a CAPTCHA that could not be solved (CAPTCHA_UNSOLVED) |
Submit an Advance job
import os
import requests
API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]
# Advance adds browser handling and may handle supported CAPTCHA challenges as part of scraping.
# Leave "package" out (or send "regular") for the standard price.
response = requests.post(
f"{API}/v1/jobs",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"urls": ["https://example.com/listing"],
"package": "advance",
"options": {"postLoadDelayMs": 3000},
},
timeout=30,
)
response.raise_for_status()
job = response.json()
packages = job["packages"]
print(f"job {job['jobId']}: {packages['regular']} Regular, {packages['advance']} Advance")
# Hosts configured to require additional Advance handling are routed as Advance.
for hostname in job.get("advanceRequiredFor", []):
print(f"{hostname} is always Advance")
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");
// Advance adds browser handling and may handle supported CAPTCHA challenges as part of scraping.
// Leave "package" out (or send "regular") for the standard price.
const response = await fetch(`${API}/v1/jobs`, {
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
urls: ["https://example.com/listing"],
package: "advance",
options: { postLoadDelayMs: 3000 },
}),
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${await response.text()}`);
}
const job = await response.json();
const { regular, advance } = job.packages;
console.log(`job ${job.jobId}: ${regular} Regular, ${advance} Advance`);
// Hosts configured to require additional Advance handling are routed as Advance.
for (const hostname of job.advanceRequiredFor ?? []) {
console.log(`${hostname} is always Advance`);
}
API="${API:-https://bankworks.info}" # your key is read from $API_KEY
# Advance adds browser handling and may handle supported CAPTCHA challenges as part of scraping.
# Leave "package" out (or send "regular") for the standard price.
curl -sS -X POST "$API/v1/jobs" \
-H "Authorization: Bearer $API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"urls": ["https://example.com/listing"],
"package": "advance",
"options": { "postLoadDelayMs": 3000 }
}' | jq '{jobId, packages, advanceRequiredFor}'
{"ok": true, "jobId": "0199…", "taskCount": 1, "deliveryDeferred": 0,
"package": "advance", "packages": {"regular": 0, "advance": 1}}
The package applies to every URL in the job. A host configured to require
additional Advance handling is routed as Advance whatever you send: a Regular job
that contains one answers
with its hostname in advanceRequiredFor, and packages
counts how many URLs went each way, so the price is never a surprise.
Use Advance only where it is needed
Send the job as Regular first. A page behind a supported CAPTCHA challenge fails with
errorCode: "CAPTCHA_REQUIRED" and is not billed; send just those
URLs again as Advance:
import os
import requests
API = os.environ.get("API", "https://bankworks.info")
JOB_ID = os.environ["JOB_ID"] # a finished Regular job
http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['API_KEY']}"
# Collect Regular URLs that returned CAPTCHA_REQUIRED. These requests were not billed.
captcha_urls, offset = [], 0
while True:
page = http.get(f"{API}/v1/jobs/{JOB_ID}",
params={"limit": 100, "offset": offset}, timeout=30)
page.raise_for_status()
tasks = page.json()["tasks"]
captcha_urls += [t["url"] for t in tasks if t.get("errorCode") == "CAPTCHA_REQUIRED"]
if len(tasks) < 100:
break
offset += 100
if not captcha_urls:
print("no Regular task returned CAPTCHA_REQUIRED")
else:
# Only those go again, as Advance; the rest of the job stays at the Regular price.
retry = http.post(f"{API}/v1/jobs",
json={"urls": captcha_urls, "package": "advance"}, timeout=30)
retry.raise_for_status()
print(f"resent {len(captcha_urls)} URL(s) as Advance: job {retry.json()['jobId']}")
const API = process.env.API ?? "https://bankworks.info";
const { API_KEY, JOB_ID } = process.env; // JOB_ID: a finished Regular job
if (!API_KEY || !JOB_ID) throw new Error("Set API_KEY and JOB_ID first");
const headers = { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json" };
// Collect Regular URLs that returned CAPTCHA_REQUIRED. These requests were not billed.
const captchaUrls = [];
for (let offset = 0; ; offset += 100) {
const page = await fetch(`${API}/v1/jobs/${JOB_ID}?limit=100&offset=${offset}`, { headers });
if (!page.ok) throw new Error(`HTTP ${page.status}: ${await page.text()}`);
const { tasks } = await page.json();
for (const task of tasks) {
if (task.errorCode === "CAPTCHA_REQUIRED") captchaUrls.push(task.url);
}
if (tasks.length < 100) break;
}
if (!captchaUrls.length) {
console.log("no Regular task returned CAPTCHA_REQUIRED");
} else {
// Only those go again, as Advance; the rest of the job stays at the Regular price.
const retry = await fetch(`${API}/v1/jobs`, {
method: "POST",
headers,
body: JSON.stringify({ urls: captchaUrls, package: "advance" }),
});
if (!retry.ok) throw new Error(`HTTP ${retry.status}: ${await retry.text()}`);
const { jobId } = await retry.json();
console.log(`resent ${captchaUrls.length} URL(s) as Advance: job ${jobId}`);
}
API="${API:-https://bankworks.info}" # your key is read from $API_KEY
# URLs from a finished Regular job with CAPTCHA_REQUIRED (first 100 tasks).
urls=$(curl -sS -H "Authorization: Bearer $API_KEY" \
"$API/v1/jobs/$JOB_ID?limit=100&offset=0" \
| jq -c '[.tasks[] | select(.errorCode == "CAPTCHA_REQUIRED") | .url]')
if [ "$urls" = "[]" ]; then
echo "no Regular task returned CAPTCHA_REQUIRED"
else
# Only those go again, as Advance.
curl -sS -X POST "$API/v1/jobs" \
-H "Authorization: Bearer $API_KEY" \
-H 'Content-Type: application/json' \
-d "{\"urls\": $urls, \"package\": \"advance\"}" | jq '{jobId, packages}'
fi
Read a job
{"ok": true,
"job": {"jobId": "…", "status": "RUNNING", "totalTasks": 20,
"doneTasks": 14, "failedTasks": 0},
"tasks": [{"taskId": "…", "url": "…", "status": "DONE",
"attempt": 1, "httpStatus": 200, "htmlBytes": 1312004}],
"limit": 100, "offset": 0}
Tasks are paginated. A job of 1,000 URLs needs ten pages at the maximum
limit of 100.
import os
import time
import requests
API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]
JOB_ID = os.environ["JOB_ID"]
POLL_SECONDS = float(os.environ.get("POLL_SECONDS", "5"))
while True:
response = requests.get(
f"{API}/v1/jobs/{JOB_ID}",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"limit": 100, "offset": 0},
timeout=30,
)
response.raise_for_status()
body = response.json()
job = body["job"]
finished = job["doneTasks"] + job["failedTasks"]
print(f"{job['doneTasks']} done, {job['failedTasks']} failed of {job['totalTasks']}")
if finished >= job["totalTasks"]:
break
time.sleep(POLL_SECONDS)
for task in body["tasks"]:
print(task["taskId"], task["status"], task.get("error") or "")
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
const JOB_ID = process.env.JOB_ID;
const POLL_SECONDS = Number(process.env.POLL_SECONDS ?? 5);
if (!API_KEY || !JOB_ID) throw new Error("Set API_KEY and JOB_ID first");
const sleep = (seconds) => new Promise((done) => setTimeout(done, seconds * 1000));
let body;
while (true) {
const response = await fetch(`${API}/v1/jobs/${JOB_ID}?limit=100&offset=0`, {
headers: { Authorization: `Bearer ${API_KEY}` },
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
body = await response.json();
const { job } = body;
const finished = job.doneTasks + job.failedTasks;
console.log(`${job.doneTasks} done, ${job.failedTasks} failed of ${job.totalTasks}`);
if (finished >= job.totalTasks) break;
await sleep(POLL_SECONDS);
}
for (const task of body.tasks) {
console.log(task.taskId, task.status, task.error ?? "");
}
while :; do
state=$(curl -sS -H "Authorization: Bearer $API_KEY" \
"$API/v1/jobs/$JOB_ID?limit=100&offset=0")
done_n=$(printf '%s' "$state" | jq -r '.job.doneTasks')
fail_n=$(printf '%s' "$state" | jq -r '.job.failedTasks')
total=$(printf '%s' "$state" | jq -r '.job.totalTasks')
echo "$done_n done, $fail_n failed of $total"
[ "$((done_n + fail_n))" -ge "$total" ] && break
sleep "${POLL_SECONDS:-5}"
done
Task status
| Status | Meaning |
|---|---|
PENDING | Waiting for a worker |
RUNNING | Being rendered right now |
DONE | HTML stored and downloadable |
FAILED | Out of attempts; error says why |
CANCELLED | Stopped before completion |
Download the HTML
- Requires the task to be
DONE; otherwise 409. X-HTML-SHA256is the digest of the stored bytes.- Served as an attachment;
?inline=1renders it instead. - Available until
resultTtlSecondselapses, then deleted.
import hashlib
import os
import requests
API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]
TASK_ID = os.environ["TASK_ID"]
response = requests.get(
f"{API}/v1/tasks/{TASK_ID}/html",
headers={"Authorization": f"Bearer {API_KEY}"},
timeout=60,
)
response.raise_for_status()
# X-HTML-SHA256 is computed over the stored bytes, so you can prove what you got.
expected = response.headers["X-HTML-SHA256"].lower()
actual = hashlib.sha256(response.content).hexdigest()
if actual != expected:
raise SystemExit(f"checksum mismatch: {actual} != {expected}")
with open(f"{TASK_ID}.html", "wb") as out:
out.write(response.content)
print(f"wrote {TASK_ID}.html ({len(response.content)} bytes, sha256 verified)")
import { createHash } from "node:crypto";
import { writeFile } from "node:fs/promises";
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
const TASK_ID = process.env.TASK_ID;
if (!API_KEY || !TASK_ID) throw new Error("Set API_KEY and TASK_ID first");
const response = await fetch(`${API}/v1/tasks/${TASK_ID}/html`, {
headers: { Authorization: `Bearer ${API_KEY}` },
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const html = Buffer.from(await response.arrayBuffer());
// X-HTML-SHA256 is computed over the stored bytes, so you can prove what you got.
const expected = response.headers.get("x-html-sha256").toLowerCase();
const actual = createHash("sha256").update(html).digest("hex");
if (actual !== expected) throw new Error(`checksum mismatch: ${actual} != ${expected}`);
await writeFile(`${TASK_ID}.html`, html);
console.log(`wrote ${TASK_ID}.html (${html.length} bytes, sha256 verified)`);
curl -sS -H "Authorization: Bearer $API_KEY" \
-o page.html -D headers.txt \
"$API/v1/tasks/$TASK_ID/html"
grep -i x-html-sha256 headers.txt
Download a screenshot
Send "screenshot": true in options and the worker also
photographs the page it rendered. The picture is the whole page, not just the
part that fits on screen, and it is taken after the wait, the cookie banner and
any CAPTCHA gate — so it shows the page you asked for, not an overlay.
blockAssets off. A picture of a
page whose images, stylesheets and fonts were blocked shows a broken page, so
asking for one loads them. Expect the request to take longer and use more
bandwidth than the same request without a screenshot.
- Defaults to full-page JPEG at quality 80. For a lossless file use
"screenshot": { "format": "png" }; for the visible area only,{ "fullPage": false }. screenshotBytesandscreenshotTypeappear on the task once it isDONE, so you know a picture exists before you fetch it.X-Screenshot-SHA256is the digest of the stored bytes.- Served as an attachment;
?inline=1displays it instead. - 404 when the request did not ask for a screenshot, and 409 while the task is not finished.
- Deleted with the rest of the result when
resultTtlSecondselapses.
import os
import time
import requests
API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}
# screenshot: true photographs the whole page. It also turns blockAssets off, because
# a picture of a page without its images and styles is not worth having.
job = requests.post(
f"{API}/v1/jobs",
headers=HEADERS,
json={"urls": ["https://example.com/"], "options": {"screenshot": True}},
timeout=30,
)
job.raise_for_status()
job_id = job.json()["jobId"]
while True:
status = requests.get(f"{API}/v1/jobs/{job_id}?limit=1", headers=HEADERS, timeout=30)
status.raise_for_status()
task = status.json()["tasks"][0]
if task["status"] in ("DONE", "FAILED"):
break
time.sleep(2)
if task["status"] != "DONE":
raise SystemExit(f"task {task['status']}: {task.get('error', '')}")
# The task says whether a picture exists, and what it is, before you fetch it.
if not task.get("screenshotBytes"):
raise SystemExit("no screenshot stored for this task")
extension = "png" if task.get("screenshotType") == "image/png" else "jpg"
image = requests.get(f"{API}/v1/tasks/{task['taskId']}/screenshot", headers=HEADERS, timeout=60)
image.raise_for_status()
name = f"{task['taskId']}.{extension}"
with open(name, "wb") as handle:
handle.write(image.content)
print(f"wrote {name} ({len(image.content)} bytes)")
import { writeFile } from "node:fs/promises";
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");
const call = async (path, init) => {
const response = await fetch(`${API}${path}`, {
...init,
headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json", ...init?.headers },
});
if (!response.ok) throw new Error(`HTTP ${response.status} on ${path}`);
return response;
};
// screenshot: true photographs the whole page. It also turns blockAssets off, because
// a picture of a page without its images and styles is not worth having.
const { jobId } = await call("/v1/jobs", {
method: "POST",
body: JSON.stringify({ urls: ["https://example.com/"], options: { screenshot: true } }),
}).then((r) => r.json());
let task;
for (;;) {
const body = await call(`/v1/jobs/${jobId}?limit=1`).then((r) => r.json());
task = body.tasks[0];
if (["DONE", "FAILED"].includes(task.status)) break;
await new Promise((resolve) => setTimeout(resolve, 2000));
}
if (task.status !== "DONE") throw new Error(`task ${task.status}: ${task.error ?? ""}`);
// The task says whether a picture exists, and what it is, before you fetch it.
if (!task.screenshotBytes) throw new Error("no screenshot stored for this task");
const extension = task.screenshotType === "image/png" ? "png" : "jpg";
const image = Buffer.from(await call(`/v1/tasks/${task.taskId}/screenshot`).then((r) => r.arrayBuffer()));
await writeFile(`${task.taskId}.${extension}`, image);
console.log(`wrote ${task.taskId}.${extension} (${image.length} bytes)`);
# Ask for a screenshot, then download the picture once the task is DONE.
JOB=$(curl -sS -X POST "$API/v1/jobs" \
-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{"urls":["https://example.com/"],"options":{"screenshot":true}}')
JOB_ID=$(printf '%s' "$JOB" | jq -r .jobId)
# Wait for the one task to finish, then take its ID.
while :; do
STATUS=$(curl -sS -H "Authorization: Bearer $API_KEY" "$API/v1/jobs/$JOB_ID?limit=1")
TASK_STATUS=$(printf '%s' "$STATUS" | jq -r '.tasks[0].status')
[ "$TASK_STATUS" = "DONE" ] || [ "$TASK_STATUS" = "FAILED" ] && break
sleep 2
done
TASK_ID=$(printf '%s' "$STATUS" | jq -r '.tasks[0].taskId')
# screenshotType tells you the extension before you fetch anything.
TYPE=$(printf '%s' "$STATUS" | jq -r '.tasks[0].screenshotType // "image/jpeg"')
EXT=jpg; [ "$TYPE" = "image/png" ] && EXT=png
curl -sS -H "Authorization: Bearer $API_KEY" \
-o "$TASK_ID.$EXT" "$API/v1/tasks/$TASK_ID/screenshot"
echo "wrote $TASK_ID.$EXT"
The HTML and the screenshot together
The HTML and the screenshot belong to the same task, so one job gives you both:
render once with "screenshot": true, then download the page from
/v1/tasks/{taskId}/html and its picture from
/v1/tasks/{taskId}/screenshot. You are charged for the one render,
not two.
import os
import time
import requests
API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}
# One request renders the page AND takes its picture. screenshot: true also turns
# blockAssets off, so the HTML you get back has the same assets the picture shows.
job = requests.post(
f"{API}/v1/jobs",
headers=HEADERS,
json={"urls": ["https://example.com/"], "options": {"screenshot": True}},
timeout=30,
)
job.raise_for_status()
job_id = job.json()["jobId"]
while True:
status = requests.get(f"{API}/v1/jobs/{job_id}?limit=1", headers=HEADERS, timeout=30)
status.raise_for_status()
task = status.json()["tasks"][0]
if task["status"] in ("DONE", "FAILED"):
break
time.sleep(2)
if task["status"] != "DONE":
raise SystemExit(f"task {task['status']}: {task.get('error', '')}")
task_id = task["taskId"]
extension = "png" if task.get("screenshotType") == "image/png" else "jpg"
# Two downloads for the one task: the HTML and the screenshot, each from its own endpoint.
html = requests.get(f"{API}/v1/tasks/{task_id}/html", headers=HEADERS, timeout=60)
html.raise_for_status()
image = requests.get(f"{API}/v1/tasks/{task_id}/screenshot", headers=HEADERS, timeout=60)
image.raise_for_status()
with open(f"{task_id}.html", "wb") as handle:
handle.write(html.content)
with open(f"{task_id}.{extension}", "wb") as handle:
handle.write(image.content)
print(
f"wrote {task_id}.html ({len(html.content)} bytes) "
f"and {task_id}.{extension} ({len(image.content)} bytes)"
)
import { writeFile } from "node:fs/promises";
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");
const call = async (path, init) => {
const response = await fetch(`${API}${path}`, {
...init,
headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json", ...init?.headers },
});
if (!response.ok) throw new Error(`HTTP ${response.status} on ${path}`);
return response;
};
// One request renders the page AND takes its picture. screenshot: true also turns
// blockAssets off, so the HTML you get back has the same assets the picture shows.
const { jobId } = await call("/v1/jobs", {
method: "POST",
body: JSON.stringify({ urls: ["https://example.com/"], options: { screenshot: true } }),
}).then((r) => r.json());
let task;
for (;;) {
const body = await call(`/v1/jobs/${jobId}?limit=1`).then((r) => r.json());
task = body.tasks[0];
if (["DONE", "FAILED"].includes(task.status)) break;
await new Promise((resolve) => setTimeout(resolve, 2000));
}
if (task.status !== "DONE") throw new Error(`task ${task.status}: ${task.error ?? ""}`);
// Two downloads for the one task: the HTML and the screenshot, each from its own
// endpoint. The task already told us the picture's type.
const extension = task.screenshotType === "image/png" ? "png" : "jpg";
const [html, image] = await Promise.all([
call(`/v1/tasks/${task.taskId}/html`).then((r) => r.arrayBuffer()),
call(`/v1/tasks/${task.taskId}/screenshot`).then((r) => r.arrayBuffer()),
]);
await writeFile(`${task.taskId}.html`, Buffer.from(html));
await writeFile(`${task.taskId}.${extension}`, Buffer.from(image));
console.log(`wrote ${task.taskId}.html (${html.byteLength} bytes) `
+ `and ${task.taskId}.${extension} (${image.byteLength} bytes)`);
# One request renders the page AND takes its picture, then download both.
# screenshot: true also turns blockAssets off, so the HTML matches the picture.
JOB=$(curl -sS -X POST "$API/v1/jobs" \
-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{"urls":["https://example.com/"],"options":{"screenshot":true}}')
JOB_ID=$(printf '%s' "$JOB" | jq -r .jobId)
while :; do
STATUS=$(curl -sS -H "Authorization: Bearer $API_KEY" "$API/v1/jobs/$JOB_ID?limit=1")
TASK_STATUS=$(printf '%s' "$STATUS" | jq -r '.tasks[0].status')
[ "$TASK_STATUS" = "DONE" ] || [ "$TASK_STATUS" = "FAILED" ] && break
sleep 2
done
[ "$TASK_STATUS" = "DONE" ] || { echo "task $TASK_STATUS"; exit 1; }
TASK_ID=$(printf '%s' "$STATUS" | jq -r '.tasks[0].taskId')
TYPE=$(printf '%s' "$STATUS" | jq -r '.tasks[0].screenshotType // "image/jpeg"')
EXT=jpg; [ "$TYPE" = "image/png" ] && EXT=png
# Two downloads for the one task: the HTML and the screenshot.
curl -sS -H "Authorization: Bearer $API_KEY" -o "$TASK_ID.html" "$API/v1/tasks/$TASK_ID/html"
curl -sS -H "Authorization: Bearer $API_KEY" -o "$TASK_ID.$EXT" "$API/v1/tasks/$TASK_ID/screenshot"
echo "wrote $TASK_ID.html and $TASK_ID.$EXT"
Batch, do not loop
Sending 20 URLs in one job is measurably faster than 20 jobs of one URL, and it costs one request against your rate limit instead of twenty. On our own measurements the batched form finished 20 pages in 344 seconds where the sequential form took roughly three times as long.
From a list of URLs to pages on disk
A complete program: give it a file with one URL per line and it does the rest.
- Reads the list, dropping blank lines and repeats (a repeat is crawled and billed twice).
- Sends it in batches of up to 1,000 URLs — one job per batch, the most one job takes.
- Waits for every job, with one status request per job rather than one per URL.
- Downloads each page that succeeded into
pages/, checking it againstX-HTML-SHA256. - Writes
results.csv: one line per URL with its status, the saved file, or the error code.
"""Crawl a list of URLs end to end: submit, wait, save every page, report.
python batch-urls.py urls.txt # one URL per line
Writes pages/<taskId>.html for each page that succeeded, and results.csv with a
line per URL (status, file, error) so failures are easy to retry.
"""
import csv
import hashlib
import os
import sys
import time
import requests
API = os.environ.get("API", "https://bankworks.info")
POLL_SECONDS = float(os.environ.get("POLL_SECONDS", "10"))
BATCH = 1000 # the most URLs one job takes
OUT = "pages"
http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['API_KEY']}"
def call(method, path, **kwargs):
response = http.request(method, f"{API}{path}", timeout=60, **kwargs)
response.raise_for_status()
return response
# 1. The list: a file, or fifty sample pages to try it out.
if len(sys.argv) > 1:
with open(sys.argv[1], encoding="utf-8") as source:
lines = source.read().splitlines()
else:
lines = [f"https://example.com/listing?page={n}" for n in range(1, 51)]
# Blank lines and repeats dropped: a repeat would be crawled and billed twice.
urls = list(dict.fromkeys(line.strip() for line in lines if line.strip()))
# 2. One job per batch of up to 1,000 URLs -- not one job per URL.
jobs = []
for start in range(0, len(urls), BATCH):
created = call("POST", "/v1/jobs", json={
"urls": urls[start:start + BATCH],
"options": {"postLoadDelayMs": 3000},
}).json()
jobs.append(created["jobId"])
print(f"job {created['jobId']}: {created['taskCount']} URL(s)")
# 3. Wait for every job. One request per job reports all of its tasks.
pending = set(jobs)
while pending:
for job_id in sorted(pending):
job = call("GET", f"/v1/jobs/{job_id}", params={"limit": 1}).json()["job"]
if job["doneTasks"] + job["failedTasks"] >= job["totalTasks"]:
pending.discard(job_id)
print(f"job {job_id} finished: {job['doneTasks']} done, {job['failedTasks']} failed")
if pending:
time.sleep(POLL_SECONDS)
# 4. Save each page that succeeded; 5. note every URL in results.csv.
os.makedirs(OUT, exist_ok=True)
saved = failed = 0
with open("results.csv", "w", newline="", encoding="utf-8") as report:
writer = csv.writer(report)
writer.writerow(["url", "status", "file", "error"])
for job_id in jobs:
offset = 0
while True: # tasks come 100 at a time
tasks = call("GET", f"/v1/jobs/{job_id}", params={"limit": 100, "offset": offset}).json()["tasks"]
for task in tasks:
if task["status"] != "DONE":
failed += 1
writer.writerow([task["url"], task["status"], "", task.get("errorCode") or task.get("error") or ""])
continue
html = call("GET", f"/v1/tasks/{task['taskId']}/html")
if hashlib.sha256(html.content).hexdigest() != html.headers["X-HTML-SHA256"].lower():
raise RuntimeError(f"checksum mismatch for {task['url']}")
path = os.path.join(OUT, f"{task['taskId']}.html")
with open(path, "wb") as out:
out.write(html.content)
saved += 1
writer.writerow([task["url"], "DONE", path, ""])
if len(tasks) < 100:
break
offset += 100
print(f"saved {saved} page(s) to {OUT}/, {failed} failed -- see results.csv")
// Crawl a list of URLs end to end: submit, wait, save every page, report.
//
// node batch-urls.mjs urls.txt # one URL per line
// Writes pages/<taskId>.html for each page that succeeded, and results.csv with a
// line per URL (status, file, error) so failures are easy to retry.
import { createHash } from "node:crypto";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");
const POLL_MS = Number(process.env.POLL_SECONDS ?? 10) * 1000;
const BATCH = 1000; // the most URLs one job takes
const OUT = "pages";
async function call(method, path, body) {
const response = await fetch(`${API}${path}`, {
method,
headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json" },
body: body ? JSON.stringify(body) : undefined,
});
if (!response.ok) throw new Error(`HTTP ${response.status} for ${path}`);
return response;
}
const sleep = (ms) => new Promise((done) => setTimeout(done, ms));
const csv = (value) => `"${String(value ?? "").replaceAll('"', '""')}"`;
// 1. The list: a file, or fifty sample pages to try it out.
const lines = process.argv[2]
? readFileSync(process.argv[2], "utf8").split(/\r?\n/)
: Array.from({ length: 50 }, (_, i) => `https://example.com/listing?page=${i + 1}`);
// Blank lines and repeats dropped: a repeat would be crawled and billed twice.
const urls = [...new Set(lines.map((line) => line.trim()).filter(Boolean))];
// 2. One job per batch of up to 1,000 URLs -- not one job per URL.
const jobs = [];
for (let start = 0; start < urls.length; start += BATCH) {
const created = await (await call("POST", "/v1/jobs", {
urls: urls.slice(start, start + BATCH),
options: { postLoadDelayMs: 3000 },
})).json();
jobs.push(created.jobId);
console.log(`job ${created.jobId}: ${created.taskCount} URL(s)`);
}
// 3. Wait for every job. One request per job reports all of its tasks.
const pending = new Set(jobs);
while (pending.size) {
for (const jobId of [...pending]) {
const { job } = await (await call("GET", `/v1/jobs/${jobId}?limit=1`)).json();
if (job.doneTasks + job.failedTasks >= job.totalTasks) {
pending.delete(jobId);
console.log(`job ${jobId} finished: ${job.doneTasks} done, ${job.failedTasks} failed`);
}
}
if (pending.size) await sleep(POLL_MS);
}
// 4. Save each page that succeeded; 5. note every URL in results.csv.
mkdirSync(OUT, { recursive: true });
const report = ["url,status,file,error"];
let saved = 0;
let failed = 0;
for (const jobId of jobs) {
for (let offset = 0; ; offset += 100) { // tasks come 100 at a time
const { tasks } = await (await call("GET", `/v1/jobs/${jobId}?limit=100&offset=${offset}`)).json();
for (const task of tasks) {
if (task.status !== "DONE") {
failed += 1;
report.push([task.url, task.status, "", task.errorCode || task.error].map(csv).join(","));
continue;
}
const html = await call("GET", `/v1/tasks/${task.taskId}/html`);
const bytes = Buffer.from(await html.arrayBuffer());
if (createHash("sha256").update(bytes).digest("hex") !== html.headers.get("x-html-sha256").toLowerCase()) {
throw new Error(`checksum mismatch for ${task.url}`);
}
const file = `${OUT}/${task.taskId}.html`;
writeFileSync(file, bytes);
saved += 1;
report.push([task.url, "DONE", file, ""].map(csv).join(","));
}
if (tasks.length < 100) break;
}
}
writeFileSync("results.csv", `${report.join("\n")}\n`);
console.log(`saved ${saved} page(s) to ${OUT}/, ${failed} failed -- see results.csv`);
#!/usr/bin/env bash
# Crawl a list of URLs end to end: submit, wait, save every page, report.
# bash batch-urls.sh urls.txt # one URL per line
set -euo pipefail
API="${API:-https://bankworks.info}" # your key is read from $API_KEY
FILE="${1:-urls.txt}"
AUTH="Authorization: Bearer $API_KEY"
mkdir -p pages
# 1. The list (fifty sample pages if there is no file yet).
[ -f "$FILE" ] || for n in $(seq 1 50); do echo "https://example.com/listing?page=$n"; done > "$FILE"
# 2. Blank lines and repeats dropped; one job per batch of up to 1,000 URLs.
# Chunked with `range` and slicing, which every jq build supports.
jobs=$(jq -R -s -c 'split("\n") | map(gsub("^\\s+|\\s+$"; "")) | map(select(length > 0)) | unique
| . as $urls | range(0; ($urls | length); 1000) as $i | $urls[$i:$i + 1000]' "$FILE" |
while read -r batch; do
curl -sS -X POST "$API/v1/jobs" -H "$AUTH" -H 'Content-Type: application/json' \
-d "{\"urls\": $batch, \"options\": {\"postLoadDelayMs\": 3000}}" | jq -r '.jobId'
done)
echo "url,status,file,error" > results.csv
for job in $jobs; do
# 3. Wait for the job.
until curl -sS -H "$AUTH" "$API/v1/jobs/$job?limit=1" |
jq -e '.job.doneTasks + .job.failedTasks >= .job.totalTasks' > /dev/null; do
sleep "${POLL_SECONDS:-10}"
done
# 4. Save each page that succeeded; 5. note every URL in results.csv.
# (The Python and Node.js versions also check each page against X-HTML-SHA256.)
offset=0
while :; do
page=$(curl -sS -H "$AUTH" "$API/v1/jobs/$job?limit=100&offset=$offset")
printf '%s' "$page" | jq -r '.tasks[] | [.taskId, .status, .url, (.errorCode // .error // "")] | @tsv' |
while IFS=$'\t' read -r task status url error; do
if [ "$status" = "DONE" ]; then
curl -sS -H "$AUTH" -o "pages/$task.html" "$API/v1/tasks/$task/html"
echo "\"$url\",DONE,pages/$task.html," >> results.csv
else
echo "\"$url\",$status,,\"$error\"" >> results.csv
fi
done
[ "$(printf '%s' "$page" | jq '.tasks | length')" -lt 100 ] && break
offset=$((offset + 100))
done
done
echo "saved $(ls pages | wc -l | tr -d ' ') page(s) to pages/ -- see results.csv"
job 0199…: 1000 URL(s)
job 0199…: 1000 URL(s)
job 0199…: 437 URL(s)
job 0199… finished: 998 done, 2 failed
job 0199… finished: 999 done, 1 failed
job 0199… finished: 436 done, 1 failed
saved 2433 page(s) to pages/, 4 failed -- see results.csv
url,status,file,error
https://example.com/p/1,DONE,pages/0199….html,
https://example.com/p/500,FAILED,,BLOCKED
To retry what failed, keep the lines of results.csv whose status is
not DONE, write their URLs to a new file and run the program on it.
Failed URLs are never billed. The error codes
say which ones are worth retrying.