Bankworks

Crawl jobs

A job is a batch of URLs. Each URL becomes one independent task, so a slow or failing page never holds up the rest of the batch.

Create a job

POST /v1/jobsjobs:create

Body

FieldTypeDefaultNotes
urlsstring[]required1–1000 entries, http or https
resultTtlSecondsinteger60060–604800. How long the HTML stays downloadable
packagestring"regular""advance" adds browser handling for more demanding pages and can attempt supported CAPTCHA challenges as part of scraping, at the Advance price. A host configured to require Advance is routed that way; the response lists it in advanceRequiredFor.
optionsobject—See below

Render options

OptionDefaultWhat it does
enableJavaScripttrueTurn off only for static pages; it is much faster
blockAssetstrueDrops images, video and SVG from the DOM. Cuts size a lot. Set false if image URLs matter to you
navigationTimeoutMs120000How long to wait for the page to load
postLoadDelayMs1000Pause after load, before reading the DOM
scrollToBottomfalseScroll to trigger lazy-loaded content
scrollTimeoutMs45000Cap on the scrolling phase
expectedText—Treat the render as failed unless this string appears. The strongest tool you have
screenshotfalseAlso take a picture of the page, downloadable from /v1/tasks/{taskId}/screenshot. Forces blockAssets off, so the request is slower and heavier
postLoadDelayMs is the option people get wrong. Pages that navigate or hydrate with JavaScript after load need 3,000–5,000 ms. With the 1,000 ms default you may capture the page mid-transition and get Execution context was destroyed instead of content.

Reject the wrong page with expectedText

A site can answer with a page that looks fine to a browser but is not the one you wanted: a consent wall, an empty search, a "something went wrong" page. Give expectedText a string that only the right page contains, such as an attribute every listing carries. A page without it is retried at no cost, and if it never appears the task ends FAILED with errorCode: "EXPECTED_TEXT_MISSING".

A page without your text is billed once. If the page loaded normally — it was not blocked and not empty — but never contained expectedText, the last attempt is charged like a successful request: the work was done, and a mistyped string must not make a crawl free. The task shows "billed": true, and its HTML stays downloadable from GET /v1/tasks/{taskId}/html so you can see what the page said. Blocked pages and timeouts are never billed.
Python
import os
import time

import requests

API = os.environ.get("API", "https://bankworks.info")
POLL_SECONDS = float(os.environ.get("POLL_SECONDS", "5"))

http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['API_KEY']}"

# Only a page that contains this text counts as a success. A consent wall, an
# empty search or a "something went wrong" page is retried; if the text never
# appears the task fails, and that last attempt is billed (its HTML is kept).
created = http.post(f"{API}/v1/jobs", json={
    "urls": ["https://example.com/cars?page=1", "https://example.com/cars?page=2"],
    "options": {"expectedText": "data-listing-id", "postLoadDelayMs": 3000},
}, timeout=30)
created.raise_for_status()
job_id = created.json()["jobId"]

while True:
    body = http.get(f"{API}/v1/jobs/{job_id}", params={"limit": 100}, timeout=30).json()
    job = body["job"]
    if job["doneTasks"] + job["failedTasks"] >= job["totalTasks"]:
        break
    time.sleep(POLL_SECONDS)

for task in body["tasks"]:
    if task["status"] == "DONE":
        print(f"ok:        {task['url']}")
    elif task.get("errorCode") == "EXPECTED_TEXT_MISSING":
        print(f"wrong page: {task['url']}")
    else:
        print(f"failed:    {task['url']} ({task.get('errorCode')})")
Node.js
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");
const POLL_MS = Number(process.env.POLL_SECONDS ?? 5) * 1000;
const headers = { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json" };

// Only a page that contains this text counts as a success. A consent wall, an
// empty search or a "something went wrong" page is retried; if the text never
// appears the task fails, and that last attempt is billed (its HTML is kept).
const created = await fetch(`${API}/v1/jobs`, {
  method: "POST",
  headers,
  body: JSON.stringify({
    urls: ["https://example.com/cars?page=1", "https://example.com/cars?page=2"],
    options: { expectedText: "data-listing-id", postLoadDelayMs: 3000 },
  }),
});
if (!created.ok) throw new Error(`HTTP ${created.status}: ${await created.text()}`);
const { jobId } = await created.json();

let body;
for (;;) {
  body = await (await fetch(`${API}/v1/jobs/${jobId}?limit=100`, { headers })).json();
  const { doneTasks, failedTasks, totalTasks } = body.job;
  if (doneTasks + failedTasks >= totalTasks) break;
  await new Promise((resolve) => setTimeout(resolve, POLL_MS));
}

for (const task of body.tasks) {
  if (task.status === "DONE") console.log(`ok:        ${task.url}`);
  else if (task.errorCode === "EXPECTED_TEXT_MISSING") console.log(`wrong page: ${task.url}`);
  else console.log(`failed:    ${task.url} (${task.errorCode})`);
}
cURL
API="${API:-https://bankworks.info}"  # your key is read from $API_KEY

# Only a page that contains this text counts as a success. A consent wall, an
# empty search or a "something went wrong" page is retried; if the text never
# appears the task fails, and that last attempt is billed (its HTML is kept).
job_id=$(curl -sS -X POST "$API/v1/jobs" \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "urls": ["https://example.com/cars?page=1", "https://example.com/cars?page=2"],
    "options": { "expectedText": "data-listing-id", "postLoadDelayMs": 3000 }
  }' | jq -r '.jobId')

while :; do
  state=$(curl -sS -H "Authorization: Bearer $API_KEY" "$API/v1/jobs/$job_id?limit=100")
  printf '%s' "$state" | jq -e '.job.doneTasks + .job.failedTasks >= .job.totalTasks' >/dev/null && break
  sleep "${POLL_SECONDS:-5}"
done

printf '%s' "$state" | jq -r '.tasks[]
  | if .status == "DONE" then "ok:        \(.url)"
    elif .errorCode == "EXPECTED_TEXT_MISSING" then "wrong page: \(.url)"
    else "failed:    \(.url) (\(.errorCode))" end'

Example

Python
import os

import requests

API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]

response = requests.post(
    f"{API}/v1/jobs",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "urls": ["https://example.com/"],
        "resultTtlSeconds": 3600,
        "options": {
            "enableJavaScript": True,
            "postLoadDelayMs": 5000,
        },
    },
    timeout=30,
)
response.raise_for_status()

job = response.json()
print(f"job {job['jobId']} with {job['taskCount']} task(s)")
Node.js
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");

const response = await fetch(`${API}/v1/jobs`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    urls: ["https://example.com/"],
    resultTtlSeconds: 3600,
    options: { enableJavaScript: true, postLoadDelayMs: 5000 },
  }),
});
if (!response.ok) {
  throw new Error(`HTTP ${response.status}: ${await response.text()}`);
}

const job = await response.json();
console.log(`job ${job.jobId} with ${job.taskCount} task(s)`);
cURL
API="${API:-https://bankworks.info}"  # your key is read from $API_KEY

curl -sS -X POST "$API/v1/jobs" \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "urls": ["https://example.com/"],
    "resultTtlSeconds": 3600,
    "options": {
      "enableJavaScript": true,
      "postLoadDelayMs": 5000
    }
  }'

https://example.com/ is a placeholder. Replace it with a page you have the right to collect.

Response

JSON
{"ok": true, "jobId": "0199…", "taskCount": 20, "deliveryDeferred": 0,
 "package": "regular", "packages": {"regular": 20, "advance": 0}}

deliveryDeferred counts tasks whose queue delivery is being retried from the durable outbox. A positive number is not an error — the work is committed and will run; it just was not handed to the queue on the first attempt.

Regular or Advance

Every request is either Regular or Advance. You choose with the top-level package field, on the same endpoint and with the same key. Leave it out for Regular. Both packages render pages in Chrome. Advance is an integrated Web Scraping API option, not a standalone CAPTCHA-solving service; site restrictions may still prevent retrieval.

RegularAdvance
Sendnothing, or "package": "regular""package": "advance"
Browser contextFresh, isolated context for the requestWorker browser context; site cookies may persist between requests
More demanding pagesStandard browser renderingAdditional browser handling, including attempts to handle supported CAPTCHA challenges encountered during scraping
Unsupported or unsolved challengeThe task fails with CAPTCHA_REQUIREDThe task may fail with CAPTCHA_UNSOLVED
PriceThe standard price per 1,000The Advance price per 1,000 (see pricing)
Not billedBlocked pages and timeoutsBlocked pages, timeouts, and a CAPTCHA that could not be solved (CAPTCHA_UNSOLVED)

Submit an Advance job

Python
import os

import requests

API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]

# Advance adds browser handling and may handle supported CAPTCHA challenges as part of scraping.
# Leave "package" out (or send "regular") for the standard price.
response = requests.post(
    f"{API}/v1/jobs",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "urls": ["https://example.com/listing"],
        "package": "advance",
        "options": {"postLoadDelayMs": 3000},
    },
    timeout=30,
)
response.raise_for_status()

job = response.json()
packages = job["packages"]
print(f"job {job['jobId']}: {packages['regular']} Regular, {packages['advance']} Advance")
# Hosts configured to require additional Advance handling are routed as Advance.
for hostname in job.get("advanceRequiredFor", []):
    print(f"{hostname} is always Advance")
Node.js
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");

// Advance adds browser handling and may handle supported CAPTCHA challenges as part of scraping.
// Leave "package" out (or send "regular") for the standard price.
const response = await fetch(`${API}/v1/jobs`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    urls: ["https://example.com/listing"],
    package: "advance",
    options: { postLoadDelayMs: 3000 },
  }),
});
if (!response.ok) {
  throw new Error(`HTTP ${response.status}: ${await response.text()}`);
}

const job = await response.json();
const { regular, advance } = job.packages;
console.log(`job ${job.jobId}: ${regular} Regular, ${advance} Advance`);
// Hosts configured to require additional Advance handling are routed as Advance.
for (const hostname of job.advanceRequiredFor ?? []) {
  console.log(`${hostname} is always Advance`);
}
cURL
API="${API:-https://bankworks.info}"  # your key is read from $API_KEY

# Advance adds browser handling and may handle supported CAPTCHA challenges as part of scraping.
# Leave "package" out (or send "regular") for the standard price.
curl -sS -X POST "$API/v1/jobs" \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "urls": ["https://example.com/listing"],
    "package": "advance",
    "options": { "postLoadDelayMs": 3000 }
  }' | jq '{jobId, packages, advanceRequiredFor}'
JSON
{"ok": true, "jobId": "0199…", "taskCount": 1, "deliveryDeferred": 0,
 "package": "advance", "packages": {"regular": 0, "advance": 1}}

The package applies to every URL in the job. A host configured to require additional Advance handling is routed as Advance whatever you send: a Regular job that contains one answers with its hostname in advanceRequiredFor, and packages counts how many URLs went each way, so the price is never a surprise.

Use Advance only where it is needed

Send the job as Regular first. A page behind a supported CAPTCHA challenge fails with errorCode: "CAPTCHA_REQUIRED" and is not billed; send just those URLs again as Advance:

Python
import os

import requests

API = os.environ.get("API", "https://bankworks.info")
JOB_ID = os.environ["JOB_ID"]  # a finished Regular job

http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['API_KEY']}"

# Collect Regular URLs that returned CAPTCHA_REQUIRED. These requests were not billed.
captcha_urls, offset = [], 0
while True:
    page = http.get(f"{API}/v1/jobs/{JOB_ID}",
                    params={"limit": 100, "offset": offset}, timeout=30)
    page.raise_for_status()
    tasks = page.json()["tasks"]
    captcha_urls += [t["url"] for t in tasks if t.get("errorCode") == "CAPTCHA_REQUIRED"]
    if len(tasks) < 100:
        break
    offset += 100

if not captcha_urls:
    print("no Regular task returned CAPTCHA_REQUIRED")
else:
    # Only those go again, as Advance; the rest of the job stays at the Regular price.
    retry = http.post(f"{API}/v1/jobs",
                      json={"urls": captcha_urls, "package": "advance"}, timeout=30)
    retry.raise_for_status()
    print(f"resent {len(captcha_urls)} URL(s) as Advance: job {retry.json()['jobId']}")
Node.js
const API = process.env.API ?? "https://bankworks.info";
const { API_KEY, JOB_ID } = process.env; // JOB_ID: a finished Regular job
if (!API_KEY || !JOB_ID) throw new Error("Set API_KEY and JOB_ID first");
const headers = { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json" };

// Collect Regular URLs that returned CAPTCHA_REQUIRED. These requests were not billed.
const captchaUrls = [];
for (let offset = 0; ; offset += 100) {
  const page = await fetch(`${API}/v1/jobs/${JOB_ID}?limit=100&offset=${offset}`, { headers });
  if (!page.ok) throw new Error(`HTTP ${page.status}: ${await page.text()}`);
  const { tasks } = await page.json();
  for (const task of tasks) {
    if (task.errorCode === "CAPTCHA_REQUIRED") captchaUrls.push(task.url);
  }
  if (tasks.length < 100) break;
}

if (!captchaUrls.length) {
  console.log("no Regular task returned CAPTCHA_REQUIRED");
} else {
  // Only those go again, as Advance; the rest of the job stays at the Regular price.
  const retry = await fetch(`${API}/v1/jobs`, {
    method: "POST",
    headers,
    body: JSON.stringify({ urls: captchaUrls, package: "advance" }),
  });
  if (!retry.ok) throw new Error(`HTTP ${retry.status}: ${await retry.text()}`);
  const { jobId } = await retry.json();
  console.log(`resent ${captchaUrls.length} URL(s) as Advance: job ${jobId}`);
}
cURL
API="${API:-https://bankworks.info}"  # your key is read from $API_KEY

# URLs from a finished Regular job with CAPTCHA_REQUIRED (first 100 tasks).
urls=$(curl -sS -H "Authorization: Bearer $API_KEY" \
  "$API/v1/jobs/$JOB_ID?limit=100&offset=0" \
  | jq -c '[.tasks[] | select(.errorCode == "CAPTCHA_REQUIRED") | .url]')

if [ "$urls" = "[]" ]; then
  echo "no Regular task returned CAPTCHA_REQUIRED"
else
  # Only those go again, as Advance.
  curl -sS -X POST "$API/v1/jobs" \
    -H "Authorization: Bearer $API_KEY" \
    -H 'Content-Type: application/json' \
    -d "{\"urls\": $urls, \"package\": \"advance\"}" | jq '{jobId, packages}'
fi

Read a job

GET /v1/jobs/{jobId}?limit=100&offset=0jobs:read
JSON
{"ok": true,
 "job": {"jobId": "…", "status": "RUNNING", "totalTasks": 20,
         "doneTasks": 14, "failedTasks": 0},
 "tasks": [{"taskId": "…", "url": "…", "status": "DONE",
            "attempt": 1, "httpStatus": 200, "htmlBytes": 1312004}],
 "limit": 100, "offset": 0}

Tasks are paginated. A job of 1,000 URLs needs ten pages at the maximum limit of 100.

Python
import os
import time

import requests

API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]
JOB_ID = os.environ["JOB_ID"]
POLL_SECONDS = float(os.environ.get("POLL_SECONDS", "5"))

while True:
    response = requests.get(
        f"{API}/v1/jobs/{JOB_ID}",
        headers={"Authorization": f"Bearer {API_KEY}"},
        params={"limit": 100, "offset": 0},
        timeout=30,
    )
    response.raise_for_status()
    body = response.json()

    job = body["job"]
    finished = job["doneTasks"] + job["failedTasks"]
    print(f"{job['doneTasks']} done, {job['failedTasks']} failed of {job['totalTasks']}")
    if finished >= job["totalTasks"]:
        break
    time.sleep(POLL_SECONDS)

for task in body["tasks"]:
    print(task["taskId"], task["status"], task.get("error") or "")
Node.js
const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
const JOB_ID = process.env.JOB_ID;
const POLL_SECONDS = Number(process.env.POLL_SECONDS ?? 5);
if (!API_KEY || !JOB_ID) throw new Error("Set API_KEY and JOB_ID first");

const sleep = (seconds) => new Promise((done) => setTimeout(done, seconds * 1000));

let body;
while (true) {
  const response = await fetch(`${API}/v1/jobs/${JOB_ID}?limit=100&offset=0`, {
    headers: { Authorization: `Bearer ${API_KEY}` },
  });
  if (!response.ok) throw new Error(`HTTP ${response.status}`);
  body = await response.json();

  const { job } = body;
  const finished = job.doneTasks + job.failedTasks;
  console.log(`${job.doneTasks} done, ${job.failedTasks} failed of ${job.totalTasks}`);
  if (finished >= job.totalTasks) break;
  await sleep(POLL_SECONDS);
}

for (const task of body.tasks) {
  console.log(task.taskId, task.status, task.error ?? "");
}
cURL
while :; do
  state=$(curl -sS -H "Authorization: Bearer $API_KEY" \
    "$API/v1/jobs/$JOB_ID?limit=100&offset=0")

  done_n=$(printf '%s' "$state" | jq -r '.job.doneTasks')
  fail_n=$(printf '%s' "$state" | jq -r '.job.failedTasks')
  total=$(printf '%s' "$state" | jq -r '.job.totalTasks')
  echo "$done_n done, $fail_n failed of $total"

  [ "$((done_n + fail_n))" -ge "$total" ] && break
  sleep "${POLL_SECONDS:-5}"
done

Task status

StatusMeaning
PENDINGWaiting for a worker
RUNNINGBeing rendered right now
DONEHTML stored and downloadable
FAILEDOut of attempts; error says why
CANCELLEDStopped before completion

Download the HTML

GET /v1/tasks/{taskId}/htmlresults:read
Python
import hashlib
import os

import requests

API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]
TASK_ID = os.environ["TASK_ID"]

response = requests.get(
    f"{API}/v1/tasks/{TASK_ID}/html",
    headers={"Authorization": f"Bearer {API_KEY}"},
    timeout=60,
)
response.raise_for_status()

# X-HTML-SHA256 is computed over the stored bytes, so you can prove what you got.
expected = response.headers["X-HTML-SHA256"].lower()
actual = hashlib.sha256(response.content).hexdigest()
if actual != expected:
    raise SystemExit(f"checksum mismatch: {actual} != {expected}")

with open(f"{TASK_ID}.html", "wb") as out:
    out.write(response.content)
print(f"wrote {TASK_ID}.html ({len(response.content)} bytes, sha256 verified)")
Node.js
import { createHash } from "node:crypto";
import { writeFile } from "node:fs/promises";

const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
const TASK_ID = process.env.TASK_ID;
if (!API_KEY || !TASK_ID) throw new Error("Set API_KEY and TASK_ID first");

const response = await fetch(`${API}/v1/tasks/${TASK_ID}/html`, {
  headers: { Authorization: `Bearer ${API_KEY}` },
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);

const html = Buffer.from(await response.arrayBuffer());

// X-HTML-SHA256 is computed over the stored bytes, so you can prove what you got.
const expected = response.headers.get("x-html-sha256").toLowerCase();
const actual = createHash("sha256").update(html).digest("hex");
if (actual !== expected) throw new Error(`checksum mismatch: ${actual} != ${expected}`);

await writeFile(`${TASK_ID}.html`, html);
console.log(`wrote ${TASK_ID}.html (${html.length} bytes, sha256 verified)`);
cURL
curl -sS -H "Authorization: Bearer $API_KEY" \
  -o page.html -D headers.txt \
  "$API/v1/tasks/$TASK_ID/html"

grep -i x-html-sha256 headers.txt

Download a screenshot

GET /v1/tasks/{taskId}/screenshotresults:read

Send "screenshot": true in options and the worker also photographs the page it rendered. The picture is the whole page, not just the part that fits on screen, and it is taken after the wait, the cookie banner and any CAPTCHA gate — so it shows the page you asked for, not an overlay.

A screenshot turns blockAssets off. A picture of a page whose images, stylesheets and fonts were blocked shows a broken page, so asking for one loads them. Expect the request to take longer and use more bandwidth than the same request without a screenshot.
Python
import os
import time

import requests

API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

# screenshot: true photographs the whole page. It also turns blockAssets off, because
# a picture of a page without its images and styles is not worth having.
job = requests.post(
    f"{API}/v1/jobs",
    headers=HEADERS,
    json={"urls": ["https://example.com/"], "options": {"screenshot": True}},
    timeout=30,
)
job.raise_for_status()
job_id = job.json()["jobId"]

while True:
    status = requests.get(f"{API}/v1/jobs/{job_id}?limit=1", headers=HEADERS, timeout=30)
    status.raise_for_status()
    task = status.json()["tasks"][0]
    if task["status"] in ("DONE", "FAILED"):
        break
    time.sleep(2)

if task["status"] != "DONE":
    raise SystemExit(f"task {task['status']}: {task.get('error', '')}")

# The task says whether a picture exists, and what it is, before you fetch it.
if not task.get("screenshotBytes"):
    raise SystemExit("no screenshot stored for this task")
extension = "png" if task.get("screenshotType") == "image/png" else "jpg"

image = requests.get(f"{API}/v1/tasks/{task['taskId']}/screenshot", headers=HEADERS, timeout=60)
image.raise_for_status()
name = f"{task['taskId']}.{extension}"
with open(name, "wb") as handle:
    handle.write(image.content)
print(f"wrote {name} ({len(image.content)} bytes)")
Node.js
import { writeFile } from "node:fs/promises";

const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");

const call = async (path, init) => {
  const response = await fetch(`${API}${path}`, {
    ...init,
    headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json", ...init?.headers },
  });
  if (!response.ok) throw new Error(`HTTP ${response.status} on ${path}`);
  return response;
};

// screenshot: true photographs the whole page. It also turns blockAssets off, because
// a picture of a page without its images and styles is not worth having.
const { jobId } = await call("/v1/jobs", {
  method: "POST",
  body: JSON.stringify({ urls: ["https://example.com/"], options: { screenshot: true } }),
}).then((r) => r.json());

let task;
for (;;) {
  const body = await call(`/v1/jobs/${jobId}?limit=1`).then((r) => r.json());
  task = body.tasks[0];
  if (["DONE", "FAILED"].includes(task.status)) break;
  await new Promise((resolve) => setTimeout(resolve, 2000));
}
if (task.status !== "DONE") throw new Error(`task ${task.status}: ${task.error ?? ""}`);

// The task says whether a picture exists, and what it is, before you fetch it.
if (!task.screenshotBytes) throw new Error("no screenshot stored for this task");
const extension = task.screenshotType === "image/png" ? "png" : "jpg";

const image = Buffer.from(await call(`/v1/tasks/${task.taskId}/screenshot`).then((r) => r.arrayBuffer()));
await writeFile(`${task.taskId}.${extension}`, image);
console.log(`wrote ${task.taskId}.${extension} (${image.length} bytes)`);
cURL
# Ask for a screenshot, then download the picture once the task is DONE.
JOB=$(curl -sS -X POST "$API/v1/jobs" \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{"urls":["https://example.com/"],"options":{"screenshot":true}}')
JOB_ID=$(printf '%s' "$JOB" | jq -r .jobId)

# Wait for the one task to finish, then take its ID.
while :; do
  STATUS=$(curl -sS -H "Authorization: Bearer $API_KEY" "$API/v1/jobs/$JOB_ID?limit=1")
  TASK_STATUS=$(printf '%s' "$STATUS" | jq -r '.tasks[0].status')
  [ "$TASK_STATUS" = "DONE" ] || [ "$TASK_STATUS" = "FAILED" ] && break
  sleep 2
done
TASK_ID=$(printf '%s' "$STATUS" | jq -r '.tasks[0].taskId')

# screenshotType tells you the extension before you fetch anything.
TYPE=$(printf '%s' "$STATUS" | jq -r '.tasks[0].screenshotType // "image/jpeg"')
EXT=jpg; [ "$TYPE" = "image/png" ] && EXT=png

curl -sS -H "Authorization: Bearer $API_KEY" \
  -o "$TASK_ID.$EXT" "$API/v1/tasks/$TASK_ID/screenshot"
echo "wrote $TASK_ID.$EXT"

The HTML and the screenshot together

The HTML and the screenshot belong to the same task, so one job gives you both: render once with "screenshot": true, then download the page from /v1/tasks/{taskId}/html and its picture from /v1/tasks/{taskId}/screenshot. You are charged for the one render, not two.

Python
import os
import time

import requests

API = os.environ.get("API", "https://bankworks.info")
API_KEY = os.environ["API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

# One request renders the page AND takes its picture. screenshot: true also turns
# blockAssets off, so the HTML you get back has the same assets the picture shows.
job = requests.post(
    f"{API}/v1/jobs",
    headers=HEADERS,
    json={"urls": ["https://example.com/"], "options": {"screenshot": True}},
    timeout=30,
)
job.raise_for_status()
job_id = job.json()["jobId"]

while True:
    status = requests.get(f"{API}/v1/jobs/{job_id}?limit=1", headers=HEADERS, timeout=30)
    status.raise_for_status()
    task = status.json()["tasks"][0]
    if task["status"] in ("DONE", "FAILED"):
        break
    time.sleep(2)

if task["status"] != "DONE":
    raise SystemExit(f"task {task['status']}: {task.get('error', '')}")

task_id = task["taskId"]
extension = "png" if task.get("screenshotType") == "image/png" else "jpg"

# Two downloads for the one task: the HTML and the screenshot, each from its own endpoint.
html = requests.get(f"{API}/v1/tasks/{task_id}/html", headers=HEADERS, timeout=60)
html.raise_for_status()
image = requests.get(f"{API}/v1/tasks/{task_id}/screenshot", headers=HEADERS, timeout=60)
image.raise_for_status()

with open(f"{task_id}.html", "wb") as handle:
    handle.write(html.content)
with open(f"{task_id}.{extension}", "wb") as handle:
    handle.write(image.content)
print(
    f"wrote {task_id}.html ({len(html.content)} bytes) "
    f"and {task_id}.{extension} ({len(image.content)} bytes)"
)
Node.js
import { writeFile } from "node:fs/promises";

const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");

const call = async (path, init) => {
  const response = await fetch(`${API}${path}`, {
    ...init,
    headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json", ...init?.headers },
  });
  if (!response.ok) throw new Error(`HTTP ${response.status} on ${path}`);
  return response;
};

// One request renders the page AND takes its picture. screenshot: true also turns
// blockAssets off, so the HTML you get back has the same assets the picture shows.
const { jobId } = await call("/v1/jobs", {
  method: "POST",
  body: JSON.stringify({ urls: ["https://example.com/"], options: { screenshot: true } }),
}).then((r) => r.json());

let task;
for (;;) {
  const body = await call(`/v1/jobs/${jobId}?limit=1`).then((r) => r.json());
  task = body.tasks[0];
  if (["DONE", "FAILED"].includes(task.status)) break;
  await new Promise((resolve) => setTimeout(resolve, 2000));
}
if (task.status !== "DONE") throw new Error(`task ${task.status}: ${task.error ?? ""}`);

// Two downloads for the one task: the HTML and the screenshot, each from its own
// endpoint. The task already told us the picture's type.
const extension = task.screenshotType === "image/png" ? "png" : "jpg";
const [html, image] = await Promise.all([
  call(`/v1/tasks/${task.taskId}/html`).then((r) => r.arrayBuffer()),
  call(`/v1/tasks/${task.taskId}/screenshot`).then((r) => r.arrayBuffer()),
]);

await writeFile(`${task.taskId}.html`, Buffer.from(html));
await writeFile(`${task.taskId}.${extension}`, Buffer.from(image));
console.log(`wrote ${task.taskId}.html (${html.byteLength} bytes) `
  + `and ${task.taskId}.${extension} (${image.byteLength} bytes)`);
cURL
# One request renders the page AND takes its picture, then download both.
# screenshot: true also turns blockAssets off, so the HTML matches the picture.
JOB=$(curl -sS -X POST "$API/v1/jobs" \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{"urls":["https://example.com/"],"options":{"screenshot":true}}')
JOB_ID=$(printf '%s' "$JOB" | jq -r .jobId)

while :; do
  STATUS=$(curl -sS -H "Authorization: Bearer $API_KEY" "$API/v1/jobs/$JOB_ID?limit=1")
  TASK_STATUS=$(printf '%s' "$STATUS" | jq -r '.tasks[0].status')
  [ "$TASK_STATUS" = "DONE" ] || [ "$TASK_STATUS" = "FAILED" ] && break
  sleep 2
done
[ "$TASK_STATUS" = "DONE" ] || { echo "task $TASK_STATUS"; exit 1; }

TASK_ID=$(printf '%s' "$STATUS" | jq -r '.tasks[0].taskId')
TYPE=$(printf '%s' "$STATUS" | jq -r '.tasks[0].screenshotType // "image/jpeg"')
EXT=jpg; [ "$TYPE" = "image/png" ] && EXT=png

# Two downloads for the one task: the HTML and the screenshot.
curl -sS -H "Authorization: Bearer $API_KEY" -o "$TASK_ID.html" "$API/v1/tasks/$TASK_ID/html"
curl -sS -H "Authorization: Bearer $API_KEY" -o "$TASK_ID.$EXT" "$API/v1/tasks/$TASK_ID/screenshot"
echo "wrote $TASK_ID.html and $TASK_ID.$EXT"

Batch, do not loop

Sending 20 URLs in one job is measurably faster than 20 jobs of one URL, and it costs one request against your rate limit instead of twenty. On our own measurements the batched form finished 20 pages in 344 seconds where the sequential form took roughly three times as long.

Rule of thumb. One job per batch of work, not one job per URL. Poll the job, not each task.

From a list of URLs to pages on disk

A complete program: give it a file with one URL per line and it does the rest.

  1. Reads the list, dropping blank lines and repeats (a repeat is crawled and billed twice).
  2. Sends it in batches of up to 1,000 URLs — one job per batch, the most one job takes.
  3. Waits for every job, with one status request per job rather than one per URL.
  4. Downloads each page that succeeded into pages/, checking it against X-HTML-SHA256.
  5. Writes results.csv: one line per URL with its status, the saved file, or the error code.
Python
"""Crawl a list of URLs end to end: submit, wait, save every page, report.

    python batch-urls.py urls.txt        # one URL per line
Writes pages/<taskId>.html for each page that succeeded, and results.csv with a
line per URL (status, file, error) so failures are easy to retry.
"""
import csv
import hashlib
import os
import sys
import time

import requests

API = os.environ.get("API", "https://bankworks.info")
POLL_SECONDS = float(os.environ.get("POLL_SECONDS", "10"))
BATCH = 1000          # the most URLs one job takes
OUT = "pages"

http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['API_KEY']}"


def call(method, path, **kwargs):
    response = http.request(method, f"{API}{path}", timeout=60, **kwargs)
    response.raise_for_status()
    return response


# 1. The list: a file, or fifty sample pages to try it out.
if len(sys.argv) > 1:
    with open(sys.argv[1], encoding="utf-8") as source:
        lines = source.read().splitlines()
else:
    lines = [f"https://example.com/listing?page={n}" for n in range(1, 51)]
# Blank lines and repeats dropped: a repeat would be crawled and billed twice.
urls = list(dict.fromkeys(line.strip() for line in lines if line.strip()))

# 2. One job per batch of up to 1,000 URLs -- not one job per URL.
jobs = []
for start in range(0, len(urls), BATCH):
    created = call("POST", "/v1/jobs", json={
        "urls": urls[start:start + BATCH],
        "options": {"postLoadDelayMs": 3000},
    }).json()
    jobs.append(created["jobId"])
    print(f"job {created['jobId']}: {created['taskCount']} URL(s)")

# 3. Wait for every job. One request per job reports all of its tasks.
pending = set(jobs)
while pending:
    for job_id in sorted(pending):
        job = call("GET", f"/v1/jobs/{job_id}", params={"limit": 1}).json()["job"]
        if job["doneTasks"] + job["failedTasks"] >= job["totalTasks"]:
            pending.discard(job_id)
            print(f"job {job_id} finished: {job['doneTasks']} done, {job['failedTasks']} failed")
    if pending:
        time.sleep(POLL_SECONDS)

# 4. Save each page that succeeded; 5. note every URL in results.csv.
os.makedirs(OUT, exist_ok=True)
saved = failed = 0
with open("results.csv", "w", newline="", encoding="utf-8") as report:
    writer = csv.writer(report)
    writer.writerow(["url", "status", "file", "error"])
    for job_id in jobs:
        offset = 0
        while True:  # tasks come 100 at a time
            tasks = call("GET", f"/v1/jobs/{job_id}", params={"limit": 100, "offset": offset}).json()["tasks"]
            for task in tasks:
                if task["status"] != "DONE":
                    failed += 1
                    writer.writerow([task["url"], task["status"], "", task.get("errorCode") or task.get("error") or ""])
                    continue
                html = call("GET", f"/v1/tasks/{task['taskId']}/html")
                if hashlib.sha256(html.content).hexdigest() != html.headers["X-HTML-SHA256"].lower():
                    raise RuntimeError(f"checksum mismatch for {task['url']}")
                path = os.path.join(OUT, f"{task['taskId']}.html")
                with open(path, "wb") as out:
                    out.write(html.content)
                saved += 1
                writer.writerow([task["url"], "DONE", path, ""])
            if len(tasks) < 100:
                break
            offset += 100

print(f"saved {saved} page(s) to {OUT}/, {failed} failed -- see results.csv")
Node.js
// Crawl a list of URLs end to end: submit, wait, save every page, report.
//
//     node batch-urls.mjs urls.txt        # one URL per line
// Writes pages/<taskId>.html for each page that succeeded, and results.csv with a
// line per URL (status, file, error) so failures are easy to retry.
import { createHash } from "node:crypto";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";

const API = process.env.API ?? "https://bankworks.info";
const API_KEY = process.env.API_KEY;
if (!API_KEY) throw new Error("Set API_KEY first");
const POLL_MS = Number(process.env.POLL_SECONDS ?? 10) * 1000;
const BATCH = 1000; // the most URLs one job takes
const OUT = "pages";

async function call(method, path, body) {
  const response = await fetch(`${API}${path}`, {
    method,
    headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json" },
    body: body ? JSON.stringify(body) : undefined,
  });
  if (!response.ok) throw new Error(`HTTP ${response.status} for ${path}`);
  return response;
}
const sleep = (ms) => new Promise((done) => setTimeout(done, ms));
const csv = (value) => `"${String(value ?? "").replaceAll('"', '""')}"`;

// 1. The list: a file, or fifty sample pages to try it out.
const lines = process.argv[2]
  ? readFileSync(process.argv[2], "utf8").split(/\r?\n/)
  : Array.from({ length: 50 }, (_, i) => `https://example.com/listing?page=${i + 1}`);
// Blank lines and repeats dropped: a repeat would be crawled and billed twice.
const urls = [...new Set(lines.map((line) => line.trim()).filter(Boolean))];

// 2. One job per batch of up to 1,000 URLs -- not one job per URL.
const jobs = [];
for (let start = 0; start < urls.length; start += BATCH) {
  const created = await (await call("POST", "/v1/jobs", {
    urls: urls.slice(start, start + BATCH),
    options: { postLoadDelayMs: 3000 },
  })).json();
  jobs.push(created.jobId);
  console.log(`job ${created.jobId}: ${created.taskCount} URL(s)`);
}

// 3. Wait for every job. One request per job reports all of its tasks.
const pending = new Set(jobs);
while (pending.size) {
  for (const jobId of [...pending]) {
    const { job } = await (await call("GET", `/v1/jobs/${jobId}?limit=1`)).json();
    if (job.doneTasks + job.failedTasks >= job.totalTasks) {
      pending.delete(jobId);
      console.log(`job ${jobId} finished: ${job.doneTasks} done, ${job.failedTasks} failed`);
    }
  }
  if (pending.size) await sleep(POLL_MS);
}

// 4. Save each page that succeeded; 5. note every URL in results.csv.
mkdirSync(OUT, { recursive: true });
const report = ["url,status,file,error"];
let saved = 0;
let failed = 0;
for (const jobId of jobs) {
  for (let offset = 0; ; offset += 100) { // tasks come 100 at a time
    const { tasks } = await (await call("GET", `/v1/jobs/${jobId}?limit=100&offset=${offset}`)).json();
    for (const task of tasks) {
      if (task.status !== "DONE") {
        failed += 1;
        report.push([task.url, task.status, "", task.errorCode || task.error].map(csv).join(","));
        continue;
      }
      const html = await call("GET", `/v1/tasks/${task.taskId}/html`);
      const bytes = Buffer.from(await html.arrayBuffer());
      if (createHash("sha256").update(bytes).digest("hex") !== html.headers.get("x-html-sha256").toLowerCase()) {
        throw new Error(`checksum mismatch for ${task.url}`);
      }
      const file = `${OUT}/${task.taskId}.html`;
      writeFileSync(file, bytes);
      saved += 1;
      report.push([task.url, "DONE", file, ""].map(csv).join(","));
    }
    if (tasks.length < 100) break;
  }
}
writeFileSync("results.csv", `${report.join("\n")}\n`);
console.log(`saved ${saved} page(s) to ${OUT}/, ${failed} failed -- see results.csv`);
cURL
#!/usr/bin/env bash
# Crawl a list of URLs end to end: submit, wait, save every page, report.
#     bash batch-urls.sh urls.txt          # one URL per line
set -euo pipefail
API="${API:-https://bankworks.info}"  # your key is read from $API_KEY
FILE="${1:-urls.txt}"
AUTH="Authorization: Bearer $API_KEY"
mkdir -p pages

# 1. The list (fifty sample pages if there is no file yet).
[ -f "$FILE" ] || for n in $(seq 1 50); do echo "https://example.com/listing?page=$n"; done > "$FILE"

# 2. Blank lines and repeats dropped; one job per batch of up to 1,000 URLs.
# Chunked with `range` and slicing, which every jq build supports.
jobs=$(jq -R -s -c 'split("\n") | map(gsub("^\\s+|\\s+$"; "")) | map(select(length > 0)) | unique
  | . as $urls | range(0; ($urls | length); 1000) as $i | $urls[$i:$i + 1000]' "$FILE" |
  while read -r batch; do
    curl -sS -X POST "$API/v1/jobs" -H "$AUTH" -H 'Content-Type: application/json' \
      -d "{\"urls\": $batch, \"options\": {\"postLoadDelayMs\": 3000}}" | jq -r '.jobId'
  done)

echo "url,status,file,error" > results.csv
for job in $jobs; do
  # 3. Wait for the job.
  until curl -sS -H "$AUTH" "$API/v1/jobs/$job?limit=1" |
        jq -e '.job.doneTasks + .job.failedTasks >= .job.totalTasks' > /dev/null; do
    sleep "${POLL_SECONDS:-10}"
  done
  # 4. Save each page that succeeded; 5. note every URL in results.csv.
  # (The Python and Node.js versions also check each page against X-HTML-SHA256.)
  offset=0
  while :; do
    page=$(curl -sS -H "$AUTH" "$API/v1/jobs/$job?limit=100&offset=$offset")
    printf '%s' "$page" | jq -r '.tasks[] | [.taskId, .status, .url, (.errorCode // .error // "")] | @tsv' |
    while IFS=$'\t' read -r task status url error; do
      if [ "$status" = "DONE" ]; then
        curl -sS -H "$AUTH" -o "pages/$task.html" "$API/v1/tasks/$task/html"
        echo "\"$url\",DONE,pages/$task.html," >> results.csv
      else
        echo "\"$url\",$status,,\"$error\"" >> results.csv
      fi
    done
    [ "$(printf '%s' "$page" | jq '.tasks | length')" -lt 100 ] && break
    offset=$((offset + 100))
  done
done
echo "saved $(ls pages | wc -l | tr -d ' ') page(s) to pages/ -- see results.csv"
Text
job 0199…: 1000 URL(s)
job 0199…: 1000 URL(s)
job 0199…: 437 URL(s)
job 0199… finished: 998 done, 2 failed
job 0199… finished: 999 done, 1 failed
job 0199… finished: 436 done, 1 failed
saved 2433 page(s) to pages/, 4 failed -- see results.csv
Text
url,status,file,error
https://example.com/p/1,DONE,pages/0199….html,
https://example.com/p/500,FAILED,,BLOCKED

To retry what failed, keep the lines of results.csv whose status is not DONE, write their URLs to a new file and run the program on it. Failed URLs are never billed. The error codes say which ones are worth retrying.