Crawler

The Search crawler reads public HTML, follows eligible links and uses configured or discovered sitemaps. It does not execute page JavaScript or sign in to private pages. For content only available after rendering or authentication, prepare an authorized content import instead.

Create a crawler and verify ownership

Open Crawlers in the selected named index, or begin with Content → Add content → Crawl website. For a new index, follow Set up Search first. Give each crawler a recognizable name and configure its origin and starting pages. An index can have multiple crawlers for different content scopes.

Complete the website ownership check using the provided verification file or meta tag. Verification is checked for the exact origin before crawling. Generating an installation prompt does not verify ownership or start a crawl.

Choose pages and extraction

Use include and exclude paths to keep the crawl focused, such as documentation under /docs/. Configure additional sitemaps when necessary.

The current per-run limits are 1–200 pages and link depth 0–8. A page limit is a safety boundary, not confirmation that the whole site was indexed. Inspect the run’s completion and skipped-page outcomes.

Extraction settings let you select the main content and title and exclude repeated navigation or other unwanted text. Use the supported simple tag, class or ID selectors. Optional section splitting creates separate results for eligible headings with existing destination anchors; it does not invent working anchors for the website.

Schedule updates

Keep the crawler manual or choose recurring days, time and timezone. Check the next scheduled run after saving. Existing daily or weekly schedules retain their cadence until you change the schedule settings.

Choose whether each crawl requires review or later eligible updates can publish automatically. The first baseline requires review. Automatic publication only applies to complete results within the configured removal threshold; partial crawls and large removals still require attention.

Review pages, changes and conflicts

A crawler has its own Pages, Activity and settings. Inspect fetched and skipped pages, then review additions, changes and removals before publishing the result. If another source supplies the same page, choose which content should win.

The page table shows what differs (title, URL, summary or text) before you open a page. The review panel places the version in search beside the crawled one and highlights changed words. Move between pages with Previous and Next or the arrow keys, choose a source with the buttons or the number keys, and the next undecided page opens automatically. To decide many pages at once, select them; the source choice appears above the table, and you can select every matching page across the list.

Publication updates that crawler’s contribution while retaining unrelated sources. The last published index remains available while the new version builds and if publication fails.

If the run reaches its limit, narrow the source, use several appropriately scoped crawlers or import a prepared snapshot. Review partial coverage explicitly before accepting a truncated result.

Understand exclusions

The crawler skips unsuitable responses and respects access and discovery rules, including robots.txt, excluded paths and noindex by default.

The Include pages marked noindex option is available when you deliberately want your own pages in website search. It does not override robots.txt or nofollow. Inspect the recorded reason for a skipped page rather than repeatedly raising limits.

An empty or unexpectedly small crawl needs investigation before publication. See Content and indexing for sources and Troubleshooting for missing pages.