Crawl Depth Configuration: Skim vs Full Analysis
Crawl Depth Configuration: Skim vs Full Analysis
When you initiate a crawl from a directory URL, you are prompted to choose a crawl depth. This controls how much work the system does and how long the crawl takes.
Crawl Depth Options
Skim Only
The system traverses the directory, discovers all product listings, and extracts surface-level data (names, categories, brief descriptions, visible metadata). No deep-scraping, scoring, or dossier generation is performed.
- Duration: Seconds
- Use when: You want to quickly assess how many listings a directory contains, or whether it's likely to yield useful products, before committing to a full run.
- Output: A raw listing count and surface data snapshot. No opportunity scores or dossiers are produced.
Full Analysis
The system runs the complete pipeline end-to-end: it skims the directory, then deep-scrapes each product's reviews, ratings, features, and pricing signals, scores each product across all four dimensions (replicability, market demand, revenue potential, competitive gaps), and generates a full dossier for the top-ranked products.
- Duration: Several minutes (varies by directory size)
- Use when: You are ready to commit to a full analysis run and want ranked results with dossiers ready to act on.
- Output: Ranked product dashboard with composite opportunity scores, and per-product dossiers including auto-drafted mission statements, suggested feature lists, and competitor weakness analysis.
Recommended Workflow
- Paste a directory URL and select Skim Only.
- Review the listing count and surface data to confirm the directory is worth analysing.
- Re-run the same URL with Full Analysis to generate the complete scored output.
This two-step approach avoids burning analysis time on directories that turn out to be too small, off-topic, or already well-covered.
Scoring Dimensions (Full Analysis Only)
Full Analysis scores each discovered product across four dimensions:
| Dimension | What It Measures |
|---|---|
| Replicability | How feasible it is to rebuild with our stack |
| Market Demand | Review volume and sentiment signals |
| Revenue Potential | Pricing tier and addressable market size |
| Competitive Gaps | Weaknesses revealed by negative reviews |
These four scores are combined into a single composite opportunity score used to rank products in the dashboard.