When a web scraper returns empty results, times out, or starts receiving blocked requests, the fastest fix is a structured diagnosis rather than random selector changes. This checklist shows how to inspect the response, validate parsing, isolate JavaScript and session issues, measure timeouts and rate limits, and record enough evidence to prevent the same failure from returning.
Overview
A web scraper can fail at several different layers: the request may not reach the intended page, the server may return a different response than expected, the HTML may not contain the data, or the parser may be looking in the wrong place. A browser-based scraper adds further variables, including navigation timing, cookies, redirects, browser version changes, and client-side rendering.
Start by separating collection into four stages:
- Request: Did the client connect, follow redirects, and receive a response?
- Response: Is the status, content type, encoding, and body what you expected?
- Rendering: If the page depends on JavaScript, did the required content appear before extraction?
- Parsing: Do selectors, regular expressions, or JSON paths still match the current structure?
Test one stage at a time. A parser cannot find content that was never downloaded, and increasing a timeout will not fix a selector that no longer matches after a website redesign. Keep a small sample of the raw response or a sanitized diagnostic snapshot so you can compare a successful run with a failed one.
What to track
1. Request and response details
Log the target URL, final URL after redirects, HTTP method, status code, response time, content type, response size, and detected character encoding. These fields quickly distinguish a missing page from an empty parser result. A successful request with a very small response body may indicate an error page, consent screen, login page, or access-control response rather than the content you intended to scrape.
For privacy and security, avoid storing credentials, full session cookies, authorization headers, or unnecessary personal data in routine logs. Store a request identifier and a redacted version of relevant headers instead.
2. Redirects, cookies, and session state
Compare the requested URL with the final URL. Unexpected redirects can send a scraper to a login route, a regional version, a consent page, or a canonical URL with different markup. Check whether your HTTP client follows redirects and whether the redirect chain changes between successful and failed runs.
Some sites return different content depending on cookies, language, region, or authentication state. If a browser succeeds but a simple HTTP client does not, compare the request method, headers, cookies, and session lifecycle. Do not assume that copying every browser header is necessary; add only the state your workflow legitimately requires and is permitted to use.
3. Selectors and extracted values
When results are empty, save the response and inspect it directly before changing code. Search for a distinctive heading, product identifier, table label, or JSON key that should be present. If the text is absent, the issue is probably upstream of the selector. If it is present, test the selector against the saved response.
Use stable attributes and semantic structure where possible. A selector based on a generated class name or a deeply nested position is more likely to break during a redesign. Track both the number of records found and the proportion of fields that are blank. A scraper that returns one record with empty fields may be failing differently from a scraper that returns zero records.
For structured pages, inspect JSON-LD, Open Graph fields, embedded state objects, and HTML tables before building a complex set of selectors. These formats can provide a more consistent extraction path when they are present. See the guide to extracting JSON-LD, Open Graph, and microdata for a focused workflow.
4. JavaScript rendering and browser waits
View the initial HTML separately from the post-render DOM. A browser may display content that was inserted later by JavaScript, while an HTTP client sees only a shell. Conversely, a browser automation script can query too early, before an API request completes.
Prefer waiting for a meaningful condition, such as a specific content container, network response, or state change, rather than relying only on a fixed sleep. In Playwright or Puppeteer, capture the page URL, title, console errors, failed requests, and a screenshot or HTML snapshot when an extraction check fails. These artifacts help identify whether the page is still loading, showing an error state, or presenting a challenge page.
5. Timeouts, retries, and rate signals
Record connection time, time to first byte, total navigation time, and the operation that timed out. A DNS or connection timeout points to a different problem than a page-load timeout or a selector wait timeout.
Use separate limits for connection, response, navigation, and extraction waits where your tools support them. Retry transient network failures with bounded exponential backoff and a maximum attempt count. Do not blindly retry every status code or parser error: repeated requests can increase load and may worsen blocking. Apply a deliberate request rate, honor applicable site rules and terms, and stop when a service indicates that access should not continue.
6. A symptom-to-cause comparison
| Symptom | Likely causes | First diagnostic step |
|---|---|---|
| Zero records, status 200 | Changed selector, JavaScript-only content, consent page, or wrong response | Save the body and search for an expected text fragment |
| Timeout before any response | DNS, connection, network, or host availability problem | Separate connection timing from page navigation timing |
| Timeout waiting for a selector | Selector changed, content is delayed, or rendering failed | Capture a screenshot, URL, and current DOM |
| Sudden 403, 429, or challenge page | Rate limit, session change, access rule, or automated traffic detection | Pause, inspect the response, and review permitted access patterns |
| Garboled accents or symbols | Incorrect character-encoding detection | Compare declared and detected encoding before parsing |
| Duplicate or partial records | Pagination, retries, unstable pages, or interrupted writes | Log page or cursor state and use a stable record identifier |
7. A reusable incident record
For each failure, capture the timestamp, job and deployment version, target URL, request ID, status, final URL, response size, content type, elapsed times, retry count, parser version, selector or extraction rule, records found, records written, and a redacted error message. Add a response sample, screenshot, or hash where appropriate. This turns “the scraper stopped working” into a reproducible incident.
Cadence and checkpoints
Use a lightweight monitoring schedule rather than waiting for a user to report missing data. For a frequently run job, compare every run with a known-good baseline. Track success rate, timeout rate, blocked-response rate, median and high-percentile duration, record count, blank-field rate, and duplicate rate.
Review these measures at least monthly for stable workflows and more often for high-value or high-frequency pipelines. A quarterly review is useful for lower-frequency crawls and technical SEO data collection. At each review:
- Run a small canary crawl against representative URLs.
- Compare record counts and key-field completeness with recent successful runs.
- Inspect the distribution of status codes and response sizes.
- Confirm that pagination, redirects, authentication, and output storage still behave as expected.
- Check dependency, browser, and deployment changes made since the last successful run.
- Review whether the request rate and collection scope remain appropriate.
Keep output validation separate from transport monitoring. A job can have a high HTTP success rate while extracting the wrong content. If your pipeline produces files for downstream systems, also validate schema, encoding, and row counts. The guide to cleaning and deduplicating scraped data can help define those checks, while the comparison of CSV, JSON, JSONL, and Parquet covers common output choices.
How to interpret changes
Look for correlated changes instead of treating every failed record as the same incident. A failure that begins immediately after a deployment suggests a code, dependency, configuration, or browser-runtime change. A gradual rise in response time may indicate increased page complexity, a slower endpoint, or an overly aggressive concurrency setting. A sharp increase in one status code suggests a response-policy or session issue, but confirm it by inspecting the body rather than relying on the code alone.
Compare a failed response with a known-good response using a diff. Check the title, canonical URL, main content container, embedded data, and pagination controls. If the page structure changed, update selectors and add a regression fixture from the new HTML. If the content moved behind an API, verify that the API request and pagination behavior are understood and permitted before adapting the workflow. For cursor- or page-based collection, use explicit checkpoints and deduplication; see how to handle API pagination.
Do not treat a blocked request as a parsing bug. Stop retries, preserve the diagnostic evidence, and review the site’s published access requirements and your organization’s authorization. The appropriate fix may be reducing frequency, narrowing scope, using an available API, or obtaining permission—not attempting to defeat an access control.
For recurring comparisons, record a baseline after every confirmed fix. Website change detection techniques can provide a useful model for alerting on meaningful HTML or content changes rather than noisy differences; see the website change detection workflow.
When to revisit
Revisit this checklist immediately after a website redesign, parser or HTTP-client upgrade, browser-version change, hosting migration, authentication change, or unexpected increase in failed records. Schedule a monthly or quarterly review even when the scraper appears healthy, because silent extraction errors can persist while requests continue returning successful status codes.
Before closing an incident, run the scraper against representative pages and confirm all of the following:
- The final URL, status, content type, and encoding are expected.
- The required content exists in the response or rendered DOM.
- Selectors or extraction paths return the expected fields.
- Pagination and retries do not create gaps or duplicates.
- Timeouts and request rates are bounded and observable.
- Output validation catches empty, malformed, or unexpectedly small results.
- Logs contain enough redacted evidence to diagnose the next failure.
Keep one known-good fixture, one deliberately failing test, and a short change log for the scraper. That small maintenance habit makes future debugging faster and gives every dependency, redesign, or deployment review a concrete checkpoint.