Blog: Skim Before You Dig — Introducing Crawl Depth Configuration
Skim Before You Dig — Introducing Crawl Depth Configuration
Released in v0.1.39
One of the most common frustrations with automated market research tools is the all-or-nothing commitment. You paste a URL, kick off a run, wait several minutes, and only then discover the directory had twelve listings — half of them irrelevant.
With v0.1.39, that changes.
Two Speeds for Two Jobs
We've introduced a crawl depth toggle at the point of initiating a crawl. Before the system does any deep analysis, you now choose how far you want it to go:
Skim Only runs in seconds. It sweeps through the directory, pulls together every listing it can find, and surfaces the basic shape of what's there — product names, categories, and top-line metadata. No scoring, no dossiers, no deep-scraping. Just a fast read of the landscape.
Full Analysis is the complete pipeline you already know: skim, then deep-scrape every product for reviews, ratings, features, and pricing signals, score across all four opportunity dimensions, and generate ready-to-use dossiers for the top performers. This takes minutes, but it produces the full ranked dashboard and everything in it.
The Intended Workflow
The two modes are designed to be used in sequence. Skim a directory first — it costs you a few seconds and tells you whether the directory is worth your time. If it looks promising, fire the full analysis. If it doesn't, move on without having burned several minutes finding that out.
This is particularly valuable when working through a batch of unfamiliar directories. A quick skim pass across several URLs helps you triage before committing the pipeline to any of them.
Why This Matters
The full analysis pipeline is thorough by design — it has to be, to produce the level of competitive intelligence that makes the dossiers useful. But that thoroughness comes with a time cost. Crawl depth configuration means that cost is now optional and deliberate rather than automatic. You decide when the depth is warranted.
This is the first step in giving users more granular control over how crawls are scoped and prioritised. More controls in this area are planned for future releases.
Crawl depth configuration is available to all users starting in v0.1.39. See the Crawl Depth Configuration feature guide for full details.