A website crawl can discover the same page under many URLs. This checklist shows how to resolve, normalize, canonicalize, and deduplicate those links while preserving meaningful page variants for technical SEO and reliable data extraction.
Overview
A raw crawl export is an observation of links, not yet a trustworthy URL inventory. The same destination may appear as a relative path, an absolute URL, a URL with a fragment, a tracking parameter, a different hostname casing, or a redirected address. If these values are counted as separate records, crawl totals become inflated and page-level analysis becomes difficult to interpret.
A normalization workflow should produce two useful results:
- A stable comparison key: a consistently formatted URL used to group apparent duplicates.
- An audit trail: the original discovered URL, redirect information, and transformation decisions retained for review.
Do not overwrite the source value during cleanup. Store the original URL alongside fields such as the resolved URL, normalized URL, canonical URL, HTTP status, redirect target, and duplicate group. This makes it possible to explain why records were merged and to adjust rules when a site’s structure changes.
A practical order of operations is: resolve relative links, parse the URL, normalize the scheme and host, remove fragments, apply an explicit parameter policy, normalize the path, optionally follow redirects, then group records by the resulting key. Canonical tags should be treated as a separate signal rather than automatically replacing the URL discovered by the crawler.
Checklist by scenario
1. Resolving links found in HTML
Resolve relative links against the page where they were found before comparing them. For example, /products/item and ../products/item may refer to the same destination when discovered from different pages. Preserve the source page as a separate field so you can identify internal-linking patterns later.
- Record the source page URL.
- Resolve relative paths against the source page’s base URL.
- Handle protocol-relative values such as
//example.com/pageusing the source scheme. - Ignore non-page schemes such as
mailto:,tel:, andjavascript:unless your project specifically analyzes them. - Decide how to handle an HTML
baseelement and document that decision.
2. Normalizing scheme, host, and port
Parse each URL with a standards-aware URL parser rather than splitting strings manually. Lowercase the hostname for comparison, because hostnames are case-insensitive. Remove a default port when it is explicitly written, such as the conventional port associated with the URL’s scheme, but retain non-default ports because they may identify a different service.
- Choose whether HTTP and HTTPS should be compared or kept as separate crawl targets.
- Define whether
www.example.comandexample.combelong to the same inventory. - Normalize internationalized domains using the behavior of your URL library, and retain the original Unicode form when it is useful for reporting.
- Preserve subdomains unless you have explicitly classified them as one property.
3. Cleaning paths and fragments
Remove the fragment, the portion after #, from the crawl comparison key in most page inventories. Fragments usually identify a location or client-side state within a document rather than a separately fetched HTML resource. Keep the original fragment in a separate column if it matters for navigation analysis.
Normalize obvious path syntax such as dot segments, so /a/../b can be compared with /b. Be cautious with trailing slashes. A server may treat /guide and /guide/ as different resources, so choose a trailing-slash rule only when supported by observed responses, site configuration, or project requirements.
4. Removing tracking parameters
Do not remove every query parameter. Some parameters select real content, language, pagination, filters, or product variants. Create an allowlist or denylist based on the site being analyzed, and record which rule was applied.
Tracking parameters are often recognizable by names used for campaign attribution or click identifiers, but parameter names vary by implementation. Treat examples as candidates for review rather than universal rules. A safe workflow is to first report parameter frequencies, inspect which values change page content, and then exclude only parameters confirmed to be non-content signals.
- Keep meaningful parameters such as page, category, format, or product identifiers when they produce distinct content.
- Remove confirmed tracking parameters from the comparison key.
- Preserve the raw query string and a parsed parameter map for auditing.
- Sort query parameters only if parameter order is not meaningful to the application.
5. Detecting redirects and canonical targets
Normalization cannot tell you whether two URLs return the same content. For that, collect response metadata. Follow redirects according to the crawl’s scope and safety rules, record each hop when possible, and store the final response URL separately from the originally discovered URL.
Extract the page’s canonical link when available, but do not treat it as proof that the page is identical to its target. Compare the canonical target with the final response URL, status code, title, content hash, and other page signals. A canonical declaration is valuable SEO data, but it is an input to analysis rather than a replacement for crawl evidence.
6. Assigning duplicate groups
Group records by the normalized comparison key first. Then create additional groups for redirect destinations, canonical targets, or content fingerprints when those relationships answer a different question. For example, two URLs can have different normalized URLs but redirect to the same destination, while two URLs can share a canonical target without returning identical content.
A useful CSV schema is:
source_url,discovered_url,resolved_url,normalized_url,final_url,
status_code,redirect_chain,canonical_url,query_parameters,
removed_parameters,content_hash,duplicate_group,normalization_notes
What to double-check
Before using a cleaned URL list for technical SEO reporting or another extraction job, review the rules that can silently change the dataset.
- Scheme policy: Decide whether HTTP-to-HTTPS transitions are a duplicate relationship, a redirect issue, or separate records.
- Host policy: Confirm whether alternate hostnames, subdomains, and localized domains are in scope.
- Query policy: Test representative parameters for pagination, sorting, filtering, sessions, and tracking.
- Encoding: Preserve valid percent-encoding and avoid decoding and re-encoding values in a way that changes their meaning.
- Case sensitivity: Lowercase the host, but do not automatically lowercase the path or parameter values.
- Redirect behavior: Distinguish temporary observations from the final destination and retain status codes.
- Canonical consistency: Flag canonical URLs that are malformed, external, blocked from the crawl, or inconsistent with the page’s final URL.
- Sample verification: Manually inspect a sample from every large duplicate group and every transformation rule.
For large crawls, calculate counts before and after each transformation. A sudden reduction after parameter removal or host merging deserves review. Export the cleaned inventory in a format suited to the next step: CSV is convenient for spreadsheet review, while JSONL or Parquet can be more practical for repeatable pipelines. See the guide to formats for scraped data when choosing an output structure.
Common mistakes
- Using string replacement instead of URL parsing: This can corrupt encoded values, ports, credentials, or query delimiters.
- Removing all query strings: This hides real page variants and can erase important SEO findings.
- Dropping the original URL: Without the raw value, you cannot audit or reverse a cleanup rule.
- Treating fragments as separate pages: This often inflates counts when links point to headings within one document.
- Assuming redirects prove equivalence: A redirect relationship is useful, but it should remain distinct from a normalized duplicate relationship.
- Trusting canonical tags without validation: Canonical data should be compared with crawl and response evidence.
- Ignoring crawl scope: A clean URL list can still include staging hosts, external domains, asset files, or session URLs if scope filters were not applied.
When implementing URL normalization in Python, JavaScript, or a crawler framework, put the rules in a tested function and record its version with every export. Include test cases for relative links, encoded characters, repeated parameters, empty values, non-default ports, fragments, internationalized domains, and redirect chains. A repeatable function is safer than a sequence of ad hoc spreadsheet edits.
When to revisit
Re-run this review before seasonal planning cycles, after a migration or redesign, when a new subdomain or locale launches, and whenever the crawl tool or URL-parsing dependency changes. Revisit the parameter policy when filters, search functionality, analytics tagging, or application routing changes. These are common points at which previously harmless assumptions begin producing new URL variants.
For each recurring crawl, compare the current inventory with the previous one using normalized URLs and duplicate groups. Look for new parameters, new hosts, unexpected scheme changes, rising redirect groups, and canonical targets that no longer resolve. Keep a short change log containing the date, scope, normalization rules, parser version, and notable exceptions.
Before acting on the next crawl, use this final checklist:
- Save the raw crawl export.
- Resolve every in-scope link against its source context.
- Apply documented scheme, host, path, fragment, and parameter rules.
- Collect redirect and canonical signals separately.
- Assign duplicate groups and inspect representative samples.
- Export both the cleaned inventory and the transformation notes.
- Compare the result with the previous crawl before making SEO or engineering changes.
This process turns a noisy list of discovered links into a defensible URL inventory: one that supports technical SEO analysis, broken-link investigation, migration checks, and repeatable data extraction without losing the evidence needed to explain each decision.