Crawl History & Run Log
Crawl History & Run Log
The Crawl History view provides a complete record of every directory crawl the system has performed. It supports audit workflows, troubleshooting, and re-running past directories.
Accessing Crawl History
Navigate to the Crawl History section from the main navigation. The view is available to all allowlisted users (glyn@agentos.com, dylan@agentos.com).
The Run List
The history list presents all past crawl runs in reverse-chronological order. Each entry shows:
| Column | Description |
|---|---|
| Directory URL | The supplier directory URL that was submitted for crawling |
| Timestamp | Date and time the crawl run was initiated |
| Products Discovered | Count of all listings found in the directory |
| Products Scored | Count of products that completed full dossier scoring |
| Duration | Total time taken for the run to complete or fail |
| Status | completed, failed, or partial |
Run Statuses
completed— the crawl finished successfully; all discovered products were processed through the scoring pipeline.partial— the crawl ran but not all discovered products were scored (e.g. due to timeouts or rate limiting on individual listings).failed— the crawl did not complete; no or very few products were processed.
Viewing Products from a Run
Click any row in the history list to open the Run Detail view. This shows the full list of products discovered during that specific crawl, including products that may not have been scored. From here you can:
- Review every listing the crawler found
- Identify products discovered but not yet scored
- Cross-reference product counts between discovered and scored
Re-running a Past Directory
If a run has a failed or partial status, you can re-submit the same directory URL using the standard crawl workflow (paste the URL into the crawl input on the main dashboard). The new run will appear as a separate entry in the history list.
Use Cases
- Audit trail — confirm which directories have been crawled and when, ensuring consistent coverage of the proptech supplier landscape.
- Troubleshooting — identify runs that failed or returned unexpectedly low product counts.
- Coverage tracking — monitor how the product database has grown over successive crawl runs of the same directory.