Collecting every record from an API requires more than repeating a request until it fails. This guide gives you a practical checklist for identifying page numbers, offsets, cursors, and continuation tokens; preserving pagination state; handling retries and duplicates; and verifying that a web scraper or scheduled data pipeline has produced a complete dataset.
Overview
API pagination divides a large result set into smaller responses. The API may use a numbered page parameter such as page=2, an offset such as offset=100, or a token returned by the previous response. Your extraction logic must understand which value advances the request and how the API signals that no more records are available.
Before writing a loop, inspect one successful response and its documentation. Identify four things:
- Record location: the JSON path containing the current batch, such as
data,items, orresults. - Page size: the requested or actual number of records returned.
- Progress marker: the next page number, next offset, cursor, or continuation token.
- Completion signal: an empty result, a boolean such as
has_more, a missing next token, or a total-count comparison.
Do not assume that a field named next is always a URL. It might be a token, a relative path, or an object containing several request parameters. Treat the response format as part of the API contract and log a redacted sample while developing your scraper.
| Pagination style | Typical request | How to stop | Main risk |
|---|---|---|---|
| Page number | ?page=3&limit=100 | No records, short page, or explicit flag | Records can shift between requests |
| Offset and limit | ?offset=200&limit=100 | Returned count is less than the limit | Offset limits or changing datasets |
| Cursor | ?cursor=eyJ... | Next cursor is null or absent | Cursor expiration or incorrect reuse |
| Continuation token | ?continuationToken=... | No continuation token in response | Token must be sent exactly as returned |
For a reliable data pipeline, store pagination state separately from the records. A checkpoint can include the endpoint, filters, page size, current page or offset, cursor, request timestamp, and number of records written. That makes a failed run diagnosable and gives you a controlled way to resume rather than starting from the beginning without knowing what was already saved.
Checklist by scenario
When the API uses page numbers
- Set an explicit page size if the API supports it, but keep it within the documented limit.
- Start with the documented first page. Some APIs begin at zero, while others begin at one.
- Request the next page only after processing or safely storing the current page.
- Stop when the API returns no records, reports that no page remains, or returns fewer records than the requested page size when that behavior is documented.
- Record the page number and item count for every request.
A page-number loop is easy to read, but it can miss or repeat records if the underlying collection changes while you are scanning it. If the API supports a stable sort order or a cursor, prefer that option for long-running extractions.
When the API uses offsets
Offset pagination usually advances by the number of records requested. Do not automatically advance by the number returned unless the API specifically requires that behavior. A response containing fewer records may indicate the end of the collection, but an unusual response can also result from filters, permissions, or a server-side limit.
offset = 0
limit = 100
while True:
response = get_json(endpoint, params={"offset": offset, "limit": limit})
records = response.get("items", [])
if not records:
break
write_records(records)
offset += len(records)
if len(records) < limit:
break
Use this pattern only when the API documentation confirms that the offset represents a record position and that advancing by the returned count is safe. If records can be inserted or removed during the run, use a stable sort key, a snapshot identifier, or cursor pagination when available.
When the API uses cursors
A cursor is normally opaque: your scraper should store it and send it back without decoding, modifying, or sorting it. The first request may omit the cursor, while later requests include the exact value returned by the previous response.
cursor = None
while True:
params = {"limit": 100}
if cursor:
params["cursor"] = cursor
response = get_json(endpoint, params=params)
records = response.get("data", [])
write_records(records)
next_cursor = response.get("next_cursor")
if not next_cursor or not records:
break
if next_cursor == cursor:
raise RuntimeError("Cursor did not advance")
cursor = next_cursor
Checking that the cursor changes protects the job from an accidental infinite loop. If a cursor expires, save the error and checkpoint rather than silently switching to page numbers. Restarting from a new cursor may create overlap, so deduplicate using a stable record identifier.
When the API uses continuation tokens or next links
Continuation tokens often appear in fields such as nextToken, continuation, or nextPageToken. Some APIs return a complete next URL instead. Prefer the returned next URL when the API explicitly provides one, because it may include additional state that is easy to omit when rebuilding query parameters.
request_url = initial_url
request_params = initial_params
while request_url:
response = get_json(request_url, params=request_params)
write_records(response.get("results", []))
request_url = response.get("next")
request_params = None
When authentication, host restrictions, or request signing apply, validate that following a returned URL remains within the intended API host. If the response returns a token rather than a URL, retain the original endpoint and send the token using the documented parameter or header.
When the extraction must resume after failure
Write each successful batch to durable storage before advancing the checkpoint. A checkpoint should be updated atomically with the batch status where possible. On restart, begin from the last confirmed position and make the write operation idempotent. For example, use the API's stable record ID as a unique key and upsert records instead of blindly appending them.
What to double-check
- Filters and sorting: Keep filters, fields, tenant identifiers, and sort parameters identical across every page request.
- Authentication: Confirm that access tokens, cookies, or signed headers remain valid for the full run. Authenticated workflows may also require CSRF or session handling; see the authenticated web scraping guide.
- Rate limits: Add a deliberate delay or concurrency limit, honor documented responses, and use retry-after guidance when provided. Do not retry every error indefinitely.
- Transient failures: Retry selected network and server errors with bounded exponential backoff. Treat malformed responses, invalid credentials, and permission errors as actionable failures.
- Duplicate detection: Track stable IDs, not just the position of a record in a page. Pagination overlap can occur after retries or changes to the source collection.
- Completeness: Compare fetched counts with a trustworthy total when available, verify that the pagination marker advanced, and inspect the first and last records in a test run.
- Output format: Preserve the source ID, retrieval timestamp, endpoint or dataset name, and relevant pagination metadata. For downstream work, review the guidance on cleaning and deduplicating scraped data.
For a scheduled job, emit a completion record containing the run ID, start and end times, request count, record count, duplicate count, retry count, and final pagination state. A job that exits successfully but cannot report what it collected is difficult to audit.
Common mistakes
- Stopping after the first response: A successful HTTP status confirms only that one request worked. It does not confirm that the dataset is complete.
- Guessing the stopping rule: An empty page is common, but some APIs provide a boolean or token instead. Use the documented signal and add a safety limit.
- Changing parameters between pages: Accidentally dropping a filter or sort order can mix unrelated records or create gaps.
- Reusing a cursor incorrectly: Cursors are usually opaque and may be tied to the original query. Do not decode or edit them.
- Retrying a write without idempotency: If the request succeeds but the client times out, a retry can produce duplicate output. Unique keys and upserts reduce this risk.
- Ignoring empty batches: A page with no records may be a valid completion, but it may also indicate a changed filter, expired session, or unexpected response shape. Log the condition and validate it.
- Hard-coding undocumented limits: Page sizes, maximum offsets, token lifetimes, and response fields can vary by endpoint. Keep them configurable and test against the API documentation.
When to revisit
Review your pagination logic whenever the API documentation, SDK, authentication flow, response schema, or rate-limit behavior changes. Also revisit it when a recurring import reports an unexpected count, a cursor expires, a scheduled run takes longer than usual, or downstream users find missing or duplicated records.
Before a seasonal planning cycle or any high-volume collection, run a controlled test against a fixed filter. Record the expected pagination behavior, sample response shape, count checks, retry settings, and output location. If the workflow has changed, compare the new run with a prior checkpoint rather than relying only on an HTTP success status.
Use this final preflight checklist:
- Identify the record array, progress marker, and completion signal.
- Confirm the first-page value and maximum page size.
- Preserve every filter and sort parameter across requests.
- Implement bounded retries, rate limiting, and clear failure states.
- Checkpoint progress and make storage idempotent.
- Track stable IDs and verify counts, duplicates, and final state.
- Test the pipeline with a small sample before collecting the full dataset.
Once these checks pass, your web scraper is better prepared to extract all API results without losing state when the source, network, or workflow changes. For related pipeline work, see the guide to scheduling reliable web scraping jobs and the practical workflow for detecting website changes with scraping.