Choosing an export format is part of designing a reliable scraping pipeline, not an afterthought. This guide compares CSV, JSON, JSONL, and Parquet for scraped data, explains the trade-offs that affect storage and downstream processing, and provides a repeatable way to choose a format for small extracts, recurring crawls, and larger analytical workloads.
Overview
The best format for scraped data depends on where the data goes next. A spreadsheet user may need a flat CSV, while an application or document store may expect JSON. A recurring crawler benefits from JSONL because each record can be written independently. An analytics pipeline may gain better scan performance and storage efficiency from Parquet.
There is no universally correct answer. The practical question is: which format preserves the data you need while keeping the next processing step simple? Consider these four formats as points on a workflow spectrum:
| Format | Best fit | Main strength | Main limitation |
|---|---|---|---|
| CSV | Flat tables, spreadsheets, simple imports | Widely compatible and easy to inspect | Weak support for nested data and types |
| JSON | API exchange and complete page records | Preserves nested objects and arrays | Large files can be awkward to stream or edit |
| JSONL | Incremental exports and recurring crawls | One independent JSON object per line | Less convenient for spreadsheet users |
| Parquet | Analytics, warehouses, and column-oriented storage | Efficient typed columns and selective reads | Needs compatible data tools to inspect and use |
For a basic HTML table, CSV may be the most convenient destination. For pages containing JSON-LD, Open Graph fields, images, arrays, or multiple variants, JSON or JSONL usually avoids unnecessary flattening. If the output will be queried repeatedly for reporting or machine learning, Parquet is often worth evaluating after the extraction and cleaning stages are stable.
Before selecting a format, define the record boundary. A record might be one product, one URL, one article, or one observation from a recurring crawl. Clear record boundaries make both exports and later deduplication easier. For related cleanup practices, see how to clean, normalize, and deduplicate scraped data.
How to estimate
You can make the format decision with a small scoring exercise rather than relying on habit. Start by listing the destination and assigning each candidate a score from 1 to 5 for the following criteria:
- Structure: Can the format represent nested fields, arrays, nulls, and different data types without awkward workarounds?
- Write pattern: Can your scraper append records safely while the crawl is still running?
- Consumer compatibility: Can the next tool, database, analyst, or application read it directly?
- Storage and transfer: Will the format remain manageable after compression and archiving?
- Query workload: Will users read complete records, or only a few columns from many records?
- Schema tolerance: Can the pipeline accommodate fields being added, removed, or left empty?
Multiply each score by its importance weight. For example, a spreadsheet handoff might give compatibility a weight of 5 and analytics performance a weight of 1. A warehouse pipeline might reverse those priorities. The result is not a benchmark; it is a transparent decision record that can be revisited when the destination changes.
A simple selection rule is useful:
- Choose CSV when the dataset is fundamentally tabular and human users need to open it in common spreadsheet or database tools.
- Choose JSON when one complete document must be exchanged or retained, especially when nested structure matters.
- Choose JSONL when records should be streamed, appended, retried, or processed independently.
- Choose Parquet when typed, column-oriented data will be analyzed repeatedly at scale.
In many production systems, the answer is a sequence rather than a single format: write JSONL during crawling, validate and normalize the records, then convert them to Parquet for analytical storage and CSV for selected business-facing extracts.
Inputs and assumptions
The quality of an export depends more on its data contract than on the file extension. Define these inputs before building the serializer:
Record shape
List required fields, optional fields, arrays, nested objects, and identifiers. A product record might include a URL, title, price, currency, availability, image URLs, and structured attributes. Flattening all attributes into CSV columns may create unstable headers when products differ. JSON or JSONL can preserve the original shape more naturally.
Type policy
Decide whether prices are numeric, dates use a consistent representation, and missing values are null, empty strings, or omitted fields. CSV does not preserve types by itself, so a value such as 00125 may be interpreted differently by different readers. Parquet can store explicit types, but the pipeline still needs a deliberate schema.
Append and recovery behavior
A long crawl should not require rewriting the entire output after every record. JSONL is well suited to append-oriented writes because each line is a separate object. CSV can also be appended when the header and column order are controlled. A single JSON array is less convenient for partial writes unless the application manages the document carefully.
Compression and storage
Estimate the uncompressed size from the number of records and average record size, then compare compressed test files rather than assuming a fixed ratio. The actual result depends on repeated text, field names, nesting, and value variation. Keep raw output separate from normalized output when reproducibility matters, and record the scraper version and collection time in metadata.
Downstream requirements
Ask whether the next step is a spreadsheet, API request, database load, search index, data warehouse, or machine-learning job. Also identify whether the consumer expects one file, a directory of partitioned files, or a stream. These requirements often decide the format more strongly than the source website does.
For technical SEO collections, a practical pattern is to retain page-level fields such as URL, status, title, canonical, and crawl timestamp in a flat table, while storing extracted JSON-LD or other nested metadata in JSONL. See the guide to extracting JSON-LD, Open Graph, and microdata for examples of data that may not fit neatly into a flat row.
Worked examples
Example 1: A one-time HTML table export
Suppose a scraper collects 10 fields from product pages and the immediate consumer wants to review the results in a spreadsheet. The records are flat, nested attributes are not required, and the export is a one-time handoff. CSV is the sensible default. Use a stable header, quote fields that contain commas or line breaks, encode text consistently, and preserve the source URL for traceability. A related workflow is covered in how to scrape HTML tables into clean CSV and JSON.
Example 2: A recurring catalog crawl
Now assume the scraper runs every night and each product can contain a variable list of images, labels, or offers. The job should continue writing even if one page fails, and a later process will validate each record. JSONL is a strong landing format. Each line can contain one product and its crawl timestamp. Failed records can be logged separately, and downstream processing can read the file incrementally rather than loading one large JSON document.
Example 3: An analytical history
Consider a long-running crawl that stores historical prices and availability for analysis. The raw landing data may remain JSONL for auditability, while cleaned snapshots are converted to Parquet with explicit columns such as product ID, observed date, price, and availability. Analysts who need only price and date can read those columns without processing every field in the original record. If a stakeholder needs a small review extract, generate CSV from the curated dataset rather than treating CSV as the system of record.
These examples illustrate a useful separation: landing format, interchange format, and analytical format do not have to be identical. Keeping those roles distinct reduces pressure to force every consumer into one compromise.
When to recalculate
Revisit the format decision whenever the inputs to the pipeline change. Recalculate your scores and run a small export test when:
- Nested fields, arrays, or new optional attributes are added.
- The destination changes from a spreadsheet to an API, database, or warehouse.
- The crawl becomes recurring, concurrent, or large enough that recovery time matters.
- Storage, transfer, or processing costs become a concern.
- Users report broken imports, lost leading zeros, inconsistent dates, or truncated fields.
- A new serialization library, compression method, cloud-storage integration, or warehouse connector enters the stack.
- You begin partitioning data by date, domain, customer, or crawl run.
Use a representative sample rather than an empty test file. Include long text, missing values, non-ASCII characters, commas, line breaks, nested arrays, duplicate records, and type-sensitive values. Measure file size, write behavior, read time for the intended query, and how easily a failed run can resume. Keep the sample schema and test results with the pipeline documentation.
As a practical default, start with CSV for simple flat exports, JSONL for resilient scraping jobs, JSON for document exchange, and Parquet for curated analytical data. Then verify the choice against the actual consumer and workload. A format is successful when it preserves meaning, supports recovery, and makes the next stage easier—not merely when it produces a smaller file.