A technical SEO crawl turns a website into a structured dataset you can inspect, compare, and act on. This guide explains how to use a web scraper to collect page status codes, metadata, canonicals, headings, indexability signals, internal links, word counts, and structured data, then organize the results into a repeatable audit workflow.
Overview
A useful SEO crawl is more than a list of URLs. It connects each URL with the technical and on-page signals needed to diagnose problems. For every page, aim to capture the requested URL, final URL, response status, content type, page title, meta description, canonical URL, heading structure, robots directives, internal links, approximate word count, and structured data.
Before crawling, define the scope. You might crawl an entire domain, a directory, a sitemap, or a sample of important templates. Record the starting URLs and the date of the crawl so that later exports can be compared. If a site contains a large number of URLs, set limits for crawl depth, page count, request rate, and allowed hosts.
Respect the site’s published crawling instructions, access controls, and operational limits. A technical SEO crawler should be diagnostic rather than disruptive. Use a clear user agent, throttle requests, handle failures gracefully, and avoid collecting data that is not needed for the audit.
A practical export schema
CSV works well for filtering and spreadsheet review, while JSON is useful for nested links and structured data. A compact page-level schema might include:
- url_requested and url_final: the starting URL and the URL after redirects.
- status_code and content_type: the HTTP result and returned resource type.
- title, meta_description, and h1: primary HTML metadata and heading fields.
- canonical: the declared canonical URL, if present.
- robots_meta and relevant response headers: indexability signals available to your crawler.
- internal_links: discovered links normalized to a consistent format.
- word_count: an approximate count based on the selected content area.
- json_ld: extracted JSON-LD blocks or a count and validation status.
- crawl_error and fetched_at: operational context for incomplete or failed requests.
Keep raw HTML or a response snapshot when your storage and permissions allow it. A raw record helps distinguish a parser problem from a page problem and makes future checks reproducible. For related cleanup practices, see how to clean, normalize, and deduplicate scraped data.
Checklist by scenario
1. Baseline technical SEO crawl
Use this checklist for a first audit or a recurring health check:
- Start with the canonical domain and follow redirects to their final destinations.
- Capture every response status, including successful pages, redirects, client errors, and server errors.
- Separate HTML documents from images, scripts, stylesheets, PDFs, and other resources.
- Extract titles, meta descriptions, H1 headings, canonical links, and robots directives.
- Count internal inbound links and list pages that link to each URL.
- Flag missing, duplicated, unusually long, or unusually short metadata for manual review rather than treating every length difference as an error.
- Store the crawl date, configuration, starting URLs, and parser version with the export.
2. Post-migration or redirect review
For a migration, preserve a list of old URLs and crawl them separately from the new site. Record the complete redirect chain, final destination, status code, and whether the destination is relevant to the original page. Then crawl the new URLs and compare titles, canonicals, headings, internal links, and indexability signals.
Pay particular attention to redirect loops, chains, unexpected hostnames, links that still point to the old domain, and pages that return a successful response but contain an error template. A before-and-after export is more useful than a single crawl because it shows what changed.
3. Template or content-change audit
When a navigation, layout, metadata template, or content system changes, group results by template or directory. Compare fields across groups to find systematic changes: every product page missing a description, every article with two H1 headings, or every newly published page lacking a canonical.
Sample representative pages manually. A scraper can identify patterns, but it cannot always determine whether a title is meaningful, whether visible content is useful, or whether a structured-data value accurately describes the page.
4. Internal-link and broken-link investigation
Build an edge list containing source URL, target URL, anchor text, and response result. Normalize fragments, remove tracking parameters where appropriate, resolve relative URLs, and classify links as internal or external. This lets you find broken internal links, orphan candidates, redirecting links, and important pages with few discoverable paths.
Do not infer that a page is orphaned from one crawl alone. JavaScript navigation, authenticated areas, XML sitemaps, and links excluded by scope can all affect discovery. Combine crawl data with known URL lists where possible.
5. Structured-data review
Extract each JSON-LD script as a separate record, including its page URL, raw value, and parsed object when valid. Also record whether the page contains other schema markup formats if your parser supports them. This creates an inventory for checking missing, duplicated, malformed, or inconsistent markup.
Structured data should be reviewed against the visible page and its purpose. Extraction tells you what is present; it does not by itself prove that the markup is correct or eligible for any search feature. The companion guide to extracting JSON-LD and schema markup covers this part of the workflow in more detail.
What to double-check
Parsing HTML is straightforward until pages behave differently from the simple cases. Before acting on crawl results, verify these areas:
- Rendered content: A plain HTTP request may receive a shell that depends on JavaScript. If important content appears only after rendering, use a browser-based approach such as Playwright and compare its output with the initial response. See the Playwright web scraping guide for the distinction.
- Canonical parsing: Multiple canonical elements, relative URLs, malformed values, or canonicals pointing to another host need separate flags. Normalize URLs before comparing them.
- Robots signals: Keep page-level meta directives distinct from response headers and from crawler access rules. They are different fields with different operational meanings.
- Word counts: Boilerplate, navigation, footers, hidden text, and repeated components can distort a raw HTML count. Define the content selector and label the result as an estimate.
- Links: Normalize trailing slashes, fragments, percent encoding, and case according to the site’s URL conventions. Preserve the original value as well as the normalized value when investigating anomalies.
- Sitemaps and robots.txt: Treat them as valuable discovery inputs and comparison sources, not as substitutes for crawling. A sitemap can contain URLs that are not linked internally, while a crawl can reveal URLs absent from it.
- Failures: Retry transient errors carefully, record timeout and connection failures separately, and never silently convert failed fetches into empty pages.
Common mistakes
- Starting without a scope. An unrestricted crawler can spend time on calendars, filters, search pages, session URLs, and infinite parameter combinations. Define allowed hosts, URL patterns, depth, and limits first.
- Reporting every difference as a defect. A missing meta description may be intentional on some page types, and a longer title is not automatically harmful. Use field-level flags to prioritize review, not to replace judgment.
- Ignoring redirects. Saving only the final URL hides redirect chains and can make migration issues difficult to diagnose. Preserve each hop or at least the full chain length and status sequence.
- Using one selector for every template. Sites often have different content structures. Maintain selectors by template where necessary and record which selector produced each extracted value.
- Mixing crawl runs. Combining data collected with different scopes or configurations creates misleading comparisons. Keep each run isolated and attach its configuration to the output.
- Discarding raw evidence. Aggregated counts are useful, but raw URLs, response details, and extracted snippets make findings explainable and repeatable.
- Ignoring operational reliability. A script that works once may fail on a large site. Add rate limits, retries with bounds, checkpointing, logging, and resumable output. The guide to reliable scheduled scraping jobs is useful when this becomes a recurring process.
When to revisit
Run a focused crawl after migrations, domain or protocol changes, major navigation updates, template releases, metadata changes, or large content imports. A broader recurring crawl can support seasonal planning and help identify gradual link, indexability, or metadata drift.
Revisit the workflow whenever your site begins relying more heavily on client-side rendering, new URL parameters, authentication, APIs, or downloadable resources. Tool and parser updates are also a reason to test a small known URL set before trusting a full export. If authenticated pages are in scope, document the permitted login flow and review the separate guidance on authenticated web scraping.
For the next crawl, use this practical sequence:
- Write the audit question and define the URL scope.
- Choose the request or browser tool that matches the site’s rendering behavior.
- Run a small test set and inspect raw responses against the parsed fields.
- Run the controlled crawl, saving errors and configuration details.
- Normalize URLs, deduplicate records, and validate key fields.
- Group findings by severity, template, directory, and likely owner.
- Manually verify high-impact examples before creating implementation tasks.
- Save the export as a baseline and schedule the next review around the next meaningful site or workflow change.
A dependable technical SEO scraper is not defined by the number of fields it extracts. It is defined by whether the data is scoped, reproducible, interpretable, and connected to a clear decision.