Crawl & Analysis Pipeline Status Tracker

Available from: v0.1.10

After you paste a directory URL and kick off a crawl, the system works through a multi-stage pipeline to discover, scrape, score, and generate dossiers for every supplier product found. The Pipeline Status Tracker shows you exactly where each crawl run is across all four of those stages — live, on the dashboard.

The Four Pipeline Stages

Every crawl run moves through the following stages in order. The status panel reflects real-time progress through each one.

1. Listings Discovered

A single count of how many supplier listings were found when the crawler skimmed the submitted directory URL. This number is populated early in the run and is typically stable before the other stages begin.

2. Products Deep-Scraped

Displayed as a fraction — e.g. 14 / 52 — showing how many of the discovered listings have been fully deep-scraped for reviews, ratings, feature descriptions, and pricing signals.

3. Products Scored

Displayed as a fraction — e.g. 9 / 52 — showing how many products have been scored across the four opportunity dimensions:

Replicability
Market demand
Revenue potential
Competitive gaps

4. Dossiers Generated

Displayed as a fraction — e.g. 6 / 52 — showing how many full dossiers (including auto-drafted mission statement, suggested feature list, and competitor weakness analysis) are complete and ready to open.

Where to Find It

The status panel appears directly on the main dashboard, below the URL submission input and above (or alongside) the ranked results table. It is visible for any crawl run that is in-progress or recently completed.

How Updates Are Delivered

The panel stays current using either polling or server-sent events (SSE) against the crawl_runs table in the database. No manual refresh is needed — stage counts increment automatically as the pipeline advances.

Reading the Panel

A typical in-progress panel looks like this:

● Crawl in progress — started 2 minutes ago

  Listings Discovered       52
  Products Deep-Scraped     14 / 52
  Products Scored            9 / 52
  Dossiers Generated         6 / 52

Once all four counts reach their totals, the run is complete and all ranked dossiers are available for review.

Tips

Dossiers become available incrementally. You don't need to wait for the full run to complete — you can drill into any dossier that shows as generated while the pipeline is still running for the remaining products.
Scored products appear in the ranked table as they complete. The leaderboard populates in real time alongside the status panel.
If a stage count stops incrementing for an extended period, check that the crawl run has not encountered an error (error states will be surfaced in a future release).