All Docs
FeaturesAgentOS Scope OutUpdated March 12, 2026

Scheduled Re-Crawl for Monitored Directories

Scheduled Re-Crawl for Monitored Directories

Available from: v0.1.27

Instead of re-running a directory crawl manually, you can mark any directory URL as monitored. The platform will automatically re-crawl it every week, flag any newly discovered products, and notify the team — so you never miss a new supplier entering the market.

How It Works

  1. Mark a directory as monitored — On any directory URL you have previously crawled, toggle it to monitored status from the dashboard.
  2. Weekly automatic re-crawl — An Inngest cron job runs once a week and re-crawls every monitored directory, skimming all listings exactly as a manual crawl would.
  3. New product detection — Any product listing that was not present during the previous crawl is flagged automatically.
  4. Dashboard badge — Newly discovered products display a "New since last crawl" badge in the ranked results table so they stand out at a glance.
  5. In-app notification — When new products are found, an in-app notification is delivered to alert the internal team.

Monitoring a Directory

  1. Run an initial crawl on a directory URL (e.g. a Kerfuffle category page).
  2. On the directory's result page, locate the Monitor toggle and enable it.
  3. The directory is now registered for weekly automated re-crawls — no further action required.

To stop automated re-crawls, disable the Monitor toggle at any time.

Dashboard — "New Since Last Crawl" Badge

After each scheduled re-crawl the dashboard updates automatically. Products that are new since the last crawl run are marked with a "New since last crawl" badge in the product list. The badge is visible in both the ranked overview and in any individual product dossier.

Badges are cleared on the next scheduled crawl cycle once the product is no longer considered new.

In-App Notifications

Whenever a scheduled re-crawl discovers one or more new products, all users on the internal allowlist receive an in-app notification. The notification includes:

  • The name of the directory that was re-crawled
  • The number of newly discovered products
  • A direct link to the filtered dashboard view showing only the new products

Technical Notes

  • Re-crawls are powered by Inngest cron jobs running on a weekly cadence.
  • Only directories explicitly marked as monitored are included in scheduled runs; unmonitored directories are unaffected.
  • The same crawl logic used for manual runs applies — review scraping, scoring, and dossier generation all execute as normal for any newly discovered product.
  • Access remains restricted to the internal allowlist (glyn@agentos.com, dylan@agentos.com).