All Docs
FeaturesNurtureHubUpdated March 21, 2026

AI Brand Setup: Website Crawler Agent

AI Brand Setup: Website Crawler Agent

NurtureHub can automatically build your agency's brand profile by crawling your existing website. Instead of manually filling in brand configuration fields, you simply provide your company URL and the AI does the rest.

Overview

The website crawler agent uses a headless browser to visit key pages on your agency website, extract and clean the text content, and send it to the AI model with a structured extraction prompt. The result is a complete draft brand profile that is ready for your review.

This means the emails NurtureHub generates for your leads will reflect your agency's actual tone, services, and local expertise from the moment you go live — with no manual writing required.

What Gets Crawled

The crawler targets the following pages on your website:

PagePurpose
HomepageOverall brand tone and key value propositions
AboutAgency history, culture, and positioning
ServicesLettings, sales, management, and other service descriptions
TeamIndividual agent names, roles, and profiles
AreasGeographic coverage and local market knowledge

The crawler extracts and cleans raw text from each page before passing it to the AI, removing navigation menus, footers, and other non-content elements.

What Gets Generated

The AI produces a draft brand profile covering all six configuration areas:

  1. Brand tone — The voice and style your agency uses (e.g. professional, friendly, authoritative).
  2. Agency overview — A summary of who you are, how long you've been operating, and what makes you different.
  3. Service descriptions — Clear descriptions of your lettings, sales, property management, and any other services you offer.
  4. Area coverage — The towns, boroughs, or postcodes your agency serves.
  5. Team profiles — Names, roles, and brief bios of your agents and staff.
  6. Local market context — Relevant information about your local property market that the AI can reference in nurture emails.

How to Use It

  1. During onboarding (or from Settings → Brand Profile), enter your agency website URL.
  2. Confirm and start the crawl.
  3. The system will crawl your site and generate your draft brand profile. This typically completes within a few minutes.
  4. Review the generated profile in the Brand Profile editor.
  5. Make any edits you need, then save and approve the profile.

Once approved, your brand profile is active and all AI-generated emails will use it.

How the Draft is Stored

The crawled content is saved as a brand source document attached to your account. This document:

  • Records what was extracted from your site at the time of the crawl.
  • Is used to populate the editable fields in your draft brand profile.
  • Can be refreshed by running a new crawl if your website content changes.

The brand source document is separate from your approved brand profile — changes to the source document do not automatically update your live profile.

Important Notes

  • The crawler only reads publicly accessible pages. Password-protected or JavaScript-heavy single-page app content may not be fully extracted.
  • The generated draft is always presented for agent review before going live — nothing is published automatically.
  • If the crawler cannot find a specific page (e.g. no dedicated /team page), the AI will generate the best profile it can from the pages it does find. You can supplement any missing sections manually in the editor.
  • Re-running the crawl will generate a new draft but will not overwrite an already-approved brand profile unless you explicitly save and approve the new draft.

Related