Regular expressions can quickly extract predictable fields from HTML, embedded JSON, and rendered page text, but they work best when used selectively. This guide provides practical regex patterns for web scraping, explains capture groups and greedy matching, and shows how to test URLs, IDs, prices, dates, phone numbers, and repeated text before putting a pattern into a Python or JavaScript pipeline.
Overview
A regex for web scraping is a compact rule that identifies text matching a structure. For example, a pattern can find every product code beginning with SKU-, capture the numeric portion of a price, or locate links in a block of page text. The pattern does not understand the meaning of a page; it only evaluates characters and their arrangement.
That distinction matters. Regular expressions are useful for small, well-defined extraction tasks, especially after you have selected the relevant HTML element or isolated a JSON string. They are less suitable for parsing a complete HTML document, handling nested markup, or interpreting complex page state. For those jobs, use an HTML parser, a JSON parser, or a browser automation tool such as Playwright or Puppeteer, then apply regex to the resulting field when a text rule is still helpful.
Before copying a pattern into a scraper, define three things:
- Target: the exact field you want, such as a URL, SKU, or displayed price.
- Boundary: what may appear immediately before and after the field.
- Output: the complete match or a specific capture group.
Capture groups are especially important. In SKU-([A-Z0-9-]+), the complete match includes SKU-, while group 1 contains only the part after the prefix. Most programming languages let you return either the full match or the captured value.
How to estimate
Choosing a scraping regex is easier when you estimate its expected coverage before optimizing its syntax. Use a small representative sample of pages and record how many valid target fields the pattern finds, how many valid fields it misses, and how many unrelated strings it captures.
A simple repeatable check is:
- Collect examples from different templates, categories, locales, or content states.
- Label each expected value as a positive example and each near-match as a negative example.
- Run the pattern against the sample in a regex tester or a short script.
- Compare the results with the labels before expanding the pattern.
- Keep the smallest pattern that covers the intended formats without accepting unrelated text.
For instance, a pattern for a price may match $19.99 but fail on 19,99 EUR or £1,250. That is not automatically a defect. It indicates that the input specification is incomplete. Decide whether the scraper should support one currency, several currencies, or the site’s displayed format, and then test against that decision.
Greedy and lazy quantifiers affect how much text a match consumes. A greedy quantifier such as .* takes as much as it can while still allowing the rest of the pattern to match. A lazy quantifier such as .*? takes as little as possible. When extracting text between two known markers, a lazy expression is often safer:
<title>(.*?)</title>
Even this example should be treated as a limited text rule rather than a complete HTML parser. It may not handle attributes, line breaks, malformed markup, or nested elements consistently. Prefer a parser for page structure and reserve regex for a field that has already been isolated.
Inputs and assumptions
URLs
A practical URL extractor can target HTTP and HTTPS links without attempting to implement every possible URL rule:
https?://[^\s"<>]+
This pattern stops at whitespace and common markup delimiters. It may include trailing punctuation from prose, so clean values after matching. If you are extracting links from HTML, selecting href attributes with an HTML parser is usually more reliable than searching the entire document.
Product IDs and SKUs
When a site uses a documented prefix and fixed character set, anchor the rule to that format:
\bSKU-([A-Z0-9]+(?:-[A-Z0-9]+)*)\b
The first capture group returns the SKU body while the full match retains the prefix. If lowercase values are valid, use a case-insensitive option or include lowercase characters explicitly. If IDs have a fixed length, a quantifier such as {8} can reduce accidental matches.
Prices
For a dollar-style value with optional thousands separators and decimal cents, one possible starting point is:
\$\s?\d{1,3}(?:,\d{3})*(?:\.\d{2})?
This is intentionally locale-specific. It does not cover every currency or number convention. A broader field rule might capture a currency symbol or code separately, but normalization should happen in application code. Do not silently convert commas and periods without knowing which convention the source uses.
Dates
For an ISO-style date, use a narrow pattern:
\b\d{4}-\d{2}-\d{2}\b
This checks the shape, not whether the date is a real calendar date. Pass the captured value to a date library for validation. For formats such as 31/12/2026, confirm the site’s day-month ordering before parsing.
Phone numbers
Phone formats vary by country, spacing, extensions, and punctuation. A deliberately permissive starting point is:
\+?[0-9][0-9 ()-]{6,}[0-9]
Use this only to find candidates. Validate and normalize candidates with rules appropriate to the expected region. A permissive expression can also match order numbers or other digit-heavy text, so surrounding context is valuable.
Repeated text
To capture repeated labels or values, use a global or find-all operation rather than changing the regex itself. For example:
<span class='tag'>([^<]+)</span>
The capture group excludes the markup and returns the text inside each matching element. In real HTML, prefer a selector such as a CSS class and then trim, decode, and normalize the extracted text.
Worked examples
Suppose a product page contains the text SKU-AB12-XL and $1,249.00. Running the SKU pattern produces a full match of SKU-AB12-XL, with the product identifier available in the capture group. Running the price pattern produces the displayed value. Store the raw value as well as a normalized representation so that later audits can distinguish source changes from conversion logic.
In Python, use a raw string for a regex containing backslashes:
import re
text = 'SKU-AB12-XL costs $1,249.00'
sku = re.search(r'\bSKU-([A-Z0-9]+(?:-[A-Z0-9]+)*)\b', text)
price = re.search(r'\$\s?\d{1,3}(?:,\d{3})*(?:\.\d{2})?', text)
sku_value = sku.group(0) if sku else None
price_value = price.group(0) if price else None
In JavaScript, use the global flag when collecting multiple results and the case-insensitive flag only when that behavior is intended:
const text = 'Links: https://example.test/a and https://example.test/b';
const urls = text.match(/https?://[^\s"<>]+/g) || [];
Language engines differ in their support for flags, Unicode properties, lookbehind, and other advanced features. If a pattern works in an online regex tester but fails in production, check the tester’s selected engine and compare it with the runtime used by your scraper. Keep a small test fixture in version control so that changes to a page template or dependency do not silently change extraction results.
When to recalculate
Revisit a web scraping regex whenever the source format changes or the output quality falls below the level your pipeline requires. Useful triggers include a redesigned page, a new locale or currency, a switch from server-rendered HTML to embedded JSON, changed ID conventions, or a new JavaScript or Python runtime.
Use a regex tester for quick experiments: paste representative positive and negative samples, select the same engine and flags as your application, inspect every capture group, and test global matching for repeated fields. Then move the confirmed pattern into automated tests. Include edge cases such as missing values, extra whitespace, HTML entities, Unicode characters, line breaks, and punctuation adjacent to URLs.
Finally, separate extraction from cleanup. First capture the source value, then trim whitespace, decode entities, normalize numbers, validate dates, and deduplicate records. This makes failures easier to diagnose and fits naturally into a broader data pipeline. For larger jobs, combine these patterns with structured extraction and change monitoring. A crawler collecting technical SEO fields, for example, may use selectors for metadata and regex only for a narrowly defined identifier or text attribute. Review the pattern after a sample crawl, log unmatched inputs, and update it from evidence rather than from a single failing page.