Quickstart
Scraping in the AI era has gotten prohibitively difficult. You currently have 2 choices when it comes to scraping a heavily protected site. You either spend a lot of time learning about bot detection and maintain a separate tool you update every time the protection gets changed, or you spend a lot of money on 3rd party scraping services and spend a little bit of time setting it up.
Shoal is an alternative to those 2 choices for founders who are not yet at a scale where they're ready to pay thousands of dollars and don't have time to become experts on bot protection. The catch is that your resource is now people, instead of time/money.
How it works
Shoal collects data through Tide browser extension instances running in real browsers called workers. Workers passively observe pages and network traffic the browser already produces, and forward structured data to your pool. Because the requests come from a real user (and because the extension never interacts with the website) bot detection or behavioral detection - past how your workers behave - are not a factor.
Step 1: Create a pool
A pool is the container for all workers and data. Sign in and click New pool from the dashboard. Give it a name and save it. The pool ID in the URL is what workers and API clients will reference.
Step 2: Add a worker
Open the Workers tab inside your pool and click Add worker. This generates a single-use invite link.
Download the latest Tide extension from the Tide releases page, load it as an unpacked extension in Chrome (chrome://extensions → Developer mode → Load unpacked), then open the extension popup, go to the Pool tab, and paste the invite link. Once the extension connects you will see the worker appear as online.
Step 3: Receive data
Workers start forwarding data immediately as they browse matching sites. There are two ways to consume it:
Webhook
Set a URL in the Webhook tab. Shoal will POST entity data to it whenever new entities are ingested. Requests carry a timestamped Shoal-Signature (HMAC-SHA256) so you can verify they came from your pool and reject replays. See Receiving Data for the verification steps.
Websockets
Connect to the WebSocket at WS /api/pool/:pool_id/firehose with an owner API token. Each message is a scraped entity as JSON, delivered in the order results arrive.
API tokens for both are managed in the Tokens tab. For a one-step setup you can generate a token and copy a ready-made configuration prompt from the Overview tab.