Scrape Error Reporting & Retry

Available from v0.1.20

When the system attempts to scrape a product listing and something goes wrong, you now get a clear, actionable error state on the dashboard — and the ability to retry without re-running the full crawl.

---

Why Scrapes Fail

Not every product URL can be scraped successfully. Common failure reasons include:

Failure Type	Description
403 Blocked	The supplier's site actively refused the request
Timeout	The request did not complete in time
No reviews found	Page loaded but contained no review data to parse
Parsing error	Page structure could not be interpreted
robots.txt block	The URL is disallowed by the site's crawl policy

---

Error State on the Dashboard

Failed products are flagged inline on the product row in the dashboard. Instead of showing empty or stale data, the row displays a visible error indicator so you can immediately see which products need attention.

Error Tooltip

Hover over the error indicator on any failed row to see:

• HTTP status code — e.g. 403, 408, 500
• Error message — a short description of what went wrong
• robots.txt flag — whether the block is due to the site's crawl policy

This information helps you judge whether a retry is likely to succeed, or whether the product is fundamentally inaccessible to automated scraping.

---

Retrying a Failed Product

Each failed product row includes a Retry button. Clicking it re-triggers the scrape for that individual product without affecting the rest of the dashboard or requiring a full directory re-crawl.

When to retry:

• The failure was a transient timeout or network issue
• You believe a previous 403 was a rate-limit that has since cleared
• A parsing error occurred during a known-unstable period

When not to retry:

• The product is flagged as blocked by robots.txt — repeated retries will not resolve a policy-level block
• A persistent 403 suggests the supplier actively blocks automated access

---

Notes

• Error state and tooltip data are stored with the product record and persist across sessions until a successful scrape replaces them.
• Retrying does not reset the product's existing dossier data — it only updates fields that a fresh successful scrape would populate.