Crawl

Read a section of a site, within your limits.

Crawl reads a start page and the pages it links to, up to the page and depth limits you set.

Free. No sign-up. No card.

Crawlnorthwind.example/docs/*
  • /docs/1,240 wordsRead
  • /docs/install860 wordsRead
  • /docs/api2,010 wordsPartial
  • /docs/changelogPast your limit
Read 3 pages · listed 1 past your limit

How it works

From one page to a whole section.

How Crawl moves from your start page.

01

Starts at your page

Kaiju reads the start page, then the pages it links to, up to your link depth.

02

Stays in scope

Only the paths you include, never the paths you skip. It stops at your page limit.

03

Returns each page

One result for each page, read like a Scrape result. Pages past your limit are listed, not read.

What you get back

One result for each page read.

Each step lights the part of the example that answers it. Pick one, or let it play.

Step 1 of 4
GET/v1/operations/{id}/results
Example
{  "operationId": "9c4b1e27…",  "state": "completed",  "highWater": 3,  "items": [    {      "id": "d4a0ad0e…",      "sequence": 1,      "kind": "crawl",      "url": "https://example.com/docs/",      "status": "ok",      "bytes": 5840,      "bodyLink": "/v1/operations/9c4b1e27…/results/d4a0ad0e…"    },    {      "id": "669adfc5…",      "sequence": 2,      "kind": "crawl",      "url": "https://example.com/docs/install",      "status": "ok",      "bytes": 7312,      "bodyLink": "/v1/operations/9c4b1e27…/results/669adfc5…"    },    {      "id": "7e3e79c3…",      "sequence": 3,      "kind": "crawl",      "url": "https://example.com/docs/api",      "status": "ok",      "bytes": 9126,      "bodyLink": "/v1/operations/9c4b1e27…/results/7e3e79c3…"    }  ],  "complete": true}

Build a request

Every option, with its default.

Change an option and the request changes with it. Run it here, free, then keep going in the playground.

End it with /* to keep the crawl under that path.

Main options
Maximum pagesDefault: 5 pages · 1 to 10
pages
Link depthDefault: 1 links · 0 to 2
links
More options13
Discovery
Find pages fromDefault: Sitemaps and links
Scope
Include pathsUp to 10 paths, each starting with /
Skip pathsUp to 10 paths, each starting with /
Include subdomainsPlanned
Planned
Ignore query for duplicatesDefault: off
Limits
Pause between pagesDefault: 0 ms · 0 to 60,000
ms
Credit limitPlannedNow: None
Planned
Content
KeepDefault: Main content
Keep only these partsUp to 100 CSS selectors
Leave out these partsUp to 100 CSS selectors
OutputDefault: Markdown
Include extracted linksDefault: off
Page accessShared by Scrape, Map and Crawl.
Open pages withDefault: Automatic
Wait after page loadDefault: 0 ms · 0 to 30,000
ms
Time limitDefault: 30 s · 5 to 120
s
Reuse a saved copy up toPlannedNow: Fresh every run
Planned: 0 to 720 h
Mobile browserPlanned
Planned

Every option has a default Change one and the request updates. Focus an option to read what it does.

POST /v1/crawl
{
  "url": "https://northwind.example/docs/",
  "includePaths": [
    "/docs/"
  ]
}

Free. No sign-up. Run it here, then keep going in the playground.

Which endpoint

Pick the endpoint for the job.

Start from what you have. Each endpoint does one job.

You know the page you need.

You send
One page address
You get
The page as Markdown, with its source
Limit
One page for each job.
See Scrape
Smallest request
POST /v1/scrape
{
  "url": "https://northwind.example/blog/launch-week"
}

You need the list of pages before you read them.

You send
A site, or a section of it
You get
A list of page addresses
Limit
Set by the service. Read GET /v1/capabilities.
See Map
Smallest request
POST /v1/map
{
  "url": "https://northwind.example",
  "maxUrls": 100
}

You need the text of many pages in one section.

You send
A start page and your limits
You get
One result for each page read
Limit
Set by the service. Read GET /v1/capabilities.
You are on this page
Smallest request
POST /v1/crawl
{
  "url": "https://northwind.example/docs/",
  "maxPages": 5,
  "maxDepth": 1
}

You need named values, such as prices, not whole pages.

You send
Up to four pages and the fields you name
You get
One record with those values
Limit
Up to 4 pages and 20 fields for each job.
See Extract
Smallest request
POST /v1/collect
// 503 capability_unavailable: not available yet

Cost and limits

Pay for each page Kaiju reads.

Draft rates. We confirm them before release.

1 credit per page read

Each page

The draft rate for each page Kaiju reads. We confirm it before release.

Limits set by the service

Each job

Read GET /v1/capabilities for the current limits. The time limit covers the whole job, not each page.

0 credits

Not charged

Pages outside your scope: not fetched, so not charged. Pages past your limit: not read, not charged, and listed for you.

Work out a whole job in the pricing estimator.

Try Crawl

Crawl a sample section now.

The playground is free and needs no sign-up.

Privacy choices

Optional website tracking

These choices control optional browser tracking on Kaiju's public website. The app asks separately for identified product analytics. Turning Analytics off here also stops optional app analytics. These choices do not change information you intentionally submit through a contact, evaluation, or meeting form.

Essential

Website delivery, security, forms, and this saved preference.

Always on