Receiving Data

There are two ways to receive data from your pool: webhooks for push-based delivery, and the firehose WebSocket for a live stream.

Multiple scrape events can fire one after another from different HTTP responses when a user visits a page. So when different scrapes are submitted they are batched together for 5 seconds (or max 20 events) to prevent overwhelming your backend with many requests. This means all jobs are received with a minimum 5 second delay.

The full API is available as an OpenAPI JSON spec.

Websocket

The firehose is a WebSocket that delivers events in real time as workers submit data. Connect with an API token that has the read:entities scope — tokens are managed in the Tokens tab.

GET /api/pool/:pool_id/firehose?token=<api_token>
Connection: Upgrade
Upgrade: websocket

The connection is read-only. Messages sent to the server are ignored.

Filtering by site

To receive events for only specific sites, pass a comma-separated sites query param listing the site IDs you want (for example instagram, amazon). The site ID is the segment of an entity name between @ and / — @instagram/post belongs to the instagram site.

GET /api/pool/:pool_id/firehose?token=<api_token>&sites=instagram,amazon

The filter is fixed for the lifetime of the connection. To change which sites you receive, reconnect with a different sites value.

Filtering happens per entity, not per event: an event carrying entities from several sites is delivered with the non-matching entities removed, and an event left with no matching entities is dropped entirely (it is never sent as an empty batch). Omit the param, or leave it empty, to receive every site.

Receiving unchanged entities

By default, a re-scrape that produces no change to an entity is not delivered. To receive these as well, pass include_unchanged=true. Matching entities arrive with "status": "unchanged", carrying the current entity data and timestamps, which lets you confirm a value is still current (for example, that a price is unchanged) without inferring it from the absence of an event.

GET /api/pool/:pool_id/firehose?token=<api_token>&include_unchanged=true

Like the site filter, this is fixed for the lifetime of the connection; reconnect to change it. It is per connection, so two consumers on the same pool can independently choose whether to receive unchanged entities.

{
  "$schema": "https://joinshoal.org/api/schemas/event.json",
  "type": "scrape.batch",
  "pool_id": "abc123",
  "events": [
    {
      "captured_at": "2026-01-01T12:00:00Z",
      "worker": { "id": "uuid", "nickname": "my-laptop" },
      "funnel": { "name": "Media Info", "site": "instagram" },
      "entities": [
        {
          "status": "new",
          "entity": {
            "_entity": "@instagram/post",
            "_id": "123",
            "title": "..."
          }
        }
      ]
    }
  ]
}

status is one of new, updated, unchanged, deleted, or restored. A single event can carry patches for multiple entities — for example a post and its author submitted together.

If the connection drops, reconnect with the same token. There is no replay however. Events that arrived while disconnected are not re-delivered, for that reason websocket connections are only recommended in environments where you can't receive requests like in development.

Webhooks

Webhooks are the recommended method of receiving data in production.

Set a URL in the Webhook tab. Shoal will POST the same JSON envelope to it for every batch of entities ingested. By default, batches containing only unchanged entities are skipped.

Idempotency

Every webhook request carries an Idempotency-Key header that is stable across retries. A failed delivery is retried with the same key, so use it to deduplicate: if you have already processed a request with a given key, acknowledge the retry without applying it again.

Idempotency-Key: wh_1234

Verifying the signature

Every request includes a Shoal-Signature header in the form:

Shoal-Signature: t=1717459200,v1=<hmac-hex>

t is the Unix timestamp (in seconds) when the request was signed. v1 is HMAC-SHA256(webhook_secret, "<t>.<body>") as lowercase hex, keyed with your webhook secret (found in the Webhook tab). The timestamp is signed into the HMAC, so it cannot be altered without invalidating the signature.

To verify a request:

Parse t and v1 out of the header.
Reject the request if t is outside your tolerance window (5 minutes is recommended). This is what prevents an attacker from replaying a captured request later.
Recompute the HMAC over "<t>.<rawBody>" and compare it to v1 using a constant-time comparison.

const TOLERANCE_SECONDS = 300;

const header = req.headers["shoal-signature"];
const { t, v1 } = Object.fromEntries(
  header.split(",").map((part) => part.split("=")),
);

const age = Math.abs(Math.floor(Date.now() / 1000) - Number(t));
if (!Number.isFinite(age) || age > TOLERANCE_SECONDS) {
  return res.status(401).end();
}

const expected = crypto
  .createHmac("sha256", webhookSecret)
  .update(`${t}.${rawBody}`)
  .digest("hex");

const ok = crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(v1));

if (!ok) {
  return res.status(401).end();
}

The timestamp window bounds replay but does not prevent a request from being replayed within that window. If you need exactly-once processing, deduplicate on the signature (or an event ID from the payload) over the tolerance window.

Compression

If you enable gzip compression in the Webhook tab, requests are sent with Content-Encoding: gzip. The signature is computed over the decompressed JSON body, so verify it against the body after decompressing. Many HTTP frameworks do not decompress request bodies automatically, so only enable this if your server (or a middleware) decodes gzipped request bodies.

Retries

Respond with any 2xx status to acknowledge delivery. Failed deliveries are retried up to 5 times; 4xx responses are not retried.

Receiving unchanged entities

Enable Include unchanged entities in the Webhook tab to also receive entities that a re-scrape left unchanged, delivered with "status": "unchanged". This is the push equivalent of the firehose include_unchanged option.

Sending a test event

While wiring up a receiver, trigger a synthetic event end-to-end instead of waiting for a real scrape. The endpoint requires the same session cookie or API key as the rest of the authenticated API. It builds a payload from a valid entity_name (one of the pool's opted-in specs, or an explicit one you pass) with fabricated field values, then fans it out through the firehose and webhook exactly like a real scrape.

POST /api/pool/:pool_id/test_event
Content-Type: application/json

{}                                    # picks an entity automatically
{ "entity_name": "@instagram/post" }  # forces a specific entity

{
  "ok": true,
  "entity_name": "@instagram/post",
  "firehose": "dispatched",
  "webhook": "dispatched"
}

webhook is "skipped" when no webhook URL is configured on the pool.