How to Scrape HTML Tables into CSV and JSON Without Losing Data
Web ScrapingHTML TablesCSVJSONData ExtractionPython

How to Scrape HTML Tables into CSV and JSON Without Losing Data

SScraper Studio
2026-11-08
7 min read

Learn how to scrape HTML tables into reliable CSV and JSON while handling headers, links, missing values, pagination, encoding, and recurring checks.

Extracting an HTML table is easy only when the markup is clean and the data is already visible. A dependable workflow must also account for headers, merged cells, nested links, missing values, multiple tables, pagination, JavaScript rendering, and character encoding. This guide shows how to scrape HTML tables into CSV and JSON with Python, how to choose an appropriate parser, and how to build checks that keep recurring exports complete and consistent.

Overview

When you extract data from a website, start by treating the table as a data structure rather than as a visual object. A browser may display a neat grid, while the underlying HTML contains several header rows, empty cells used for layout, links that hold the real identifiers, or a table that is populated only after JavaScript runs.

A reliable process has five stages:

  1. Inspect the page and confirm that the content is inside a real <table> element.
  2. Identify the table, columns, row boundaries, and fields hidden inside links or other elements.
  3. Parse the HTML while preserving missing values and meaningful text.
  4. Normalize the records into a consistent schema.
  5. Validate the result before writing CSV or JSON.

For simple, server-rendered pages, Python with Requests and BeautifulSoup is often sufficient. Pandas can be useful when the table is conventional and you want a quick dataframe export. If the table appears only after interaction or a network request, use a browser automation tool such as Playwright, then pass the rendered HTML to a parser. This distinction is covered in more detail in the practical Playwright guide.

Page conditionPractical starting pointPrimary concern
Static table with regular columnsRequests plus BeautifulSoupCorrect headers and whitespace
Several conventional tablespandas read_htmlSelecting the intended table
Table rendered after page loadPlaywright or PuppeteerWaiting for the table and pagination
Large recurring crawlScrapy or a structured pipelineRetries, validation, and change tracking

What to track

Confirm the real table structure

Use your browser’s developer tools to inspect the page source and the live DOM. Look for the table’s selector, its thead, tbody, and tr elements, and any classes that distinguish it from layout tables. Also check whether the page contains several tables, such as a summary table followed by a detailed results table.

Before writing code, record a small field map:

  • Output field: the name you want in CSV or JSON.
  • Source cell: the column position or selector.
  • Value type: text, number, date, URL, or identifier.
  • Empty-value rule: preserve as an empty string, null, or a documented placeholder.
  • Normalization rule: trim whitespace, standardize dates, or remove display-only symbols.

A reusable BeautifulSoup pattern

The following example extracts a known table, uses the first row as a header, and writes both CSV and JSON. Adjust the selector and field rules to match the target page rather than assuming every table has identical markup.

import csv
import json
import requests
from bs4 import BeautifulSoup

url = "https://example.com/report"
response = requests.get(url, timeout=30)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
table = soup.select_one("table.report")
if table is None:
    raise ValueError("Expected report table was not found")

rows = []
for tr in table.select("tr"):
    cells = tr.select("th, td")
    values = [cell.get_text(" ", strip=True) for cell in cells]
    if values:
        rows.append(values)

if not rows:
    raise ValueError("The table contained no rows")

headers = rows[0]
records = [dict(zip(headers, row)) for row in rows[1:]]

with open("report.csv", "w", newline="", encoding="utf-8-sig") as file:
    writer = csv.DictWriter(file, fieldnames=headers, extrasaction="ignore")
    writer.writeheader()
    writer.writerows(records)

with open("report.json", "w", encoding="utf-8") as file:
    json.dump(records, file, ensure_ascii=False, indent=2)

This basic pattern is intentionally conservative. In production, verify that every data row has the expected number of cells before using zip. A row with a missing cell can otherwise shift values into the wrong columns or silently discard extra cells.

The visible text is not always the complete value. A product name may link to a detail page, or a record number may be stored in an href attribute. Extract both when the link is useful:

link = cell.select_one("a")
name = link.get_text(" ", strip=True) if link else cell.get_text(" ", strip=True)
detail_url = link.get("href") if link else None

Store these as separate fields, such as name and detail_url. Resolve relative URLs against the page URL when necessary. For recurring data extraction, preserving the original link makes later verification much easier.

Preserve meaning in difficult cells

rowspan and colspan mean that the number of visible cells may differ from the number of logical columns. A parser that simply reads each row can produce misaligned records. For these tables, either use a library that expands spans, write a grid-building routine, or map cells by stable labels instead of position.

Do not automatically convert every blank to zero or every dash to a number. An empty cell, a value of zero, and a value marked “not available” may represent different states. Preserve the source meaning and document any conversion in the export.

Cadence and checkpoints

If you scrape a report, catalog, schedule, or research table more than once, treat each run as a checkpoint. Save the retrieval time, source URL, parser version, row count, and output file together with the extracted records. These fields make it possible to tell whether a change came from the website or from your code.

A monthly or quarterly schedule is suitable for many slowly changing tables, while operational data may require a shorter interval. Choose the cadence from the business need and the expected update pattern, not from the parser alone. Respect the site’s published access instructions, avoid unnecessary requests, and use caching where it is appropriate.

At each run, check:

  • The HTTP response and final URL are expected.
  • The target table still exists and contains rows.
  • The header names have not changed unexpectedly.
  • The row count is within a reasonable range of the previous run.
  • Required fields are present and valid.
  • Pagination or “load more” controls were processed if applicable.
  • The output can be opened as valid CSV and parsed as valid JSON.

For pages with pagination, identify whether the next page uses a numbered URL, a cursor, a form submission, or a JavaScript request. Set a clear stopping condition. A scraper that follows every link on a page can accidentally collect navigation or duplicate records.

CSV and JSON output choices

CSV is convenient for spreadsheets and simple data exchanges, but it is a flat format. Use one column per scalar value and quote fields that contain commas, line breaks, or quotation marks. Always specify an encoding and newline behavior when writing files. UTF-8 is a sensible default; an application that expects a byte-order mark may require a compatible UTF-8 variant.

JSON is better suited to nested data, links, and metadata. A practical record can include the extracted fields alongside source information:

{
  "name": "Example item",
  "status": "Active",
  "detail_url": "https://example.com/items/1",
  "source_url": "https://example.com/report",
  "retrieved_at": "2026-01-15T10:30:00Z"
}

Keep the schema stable between runs. If a site adds a column, add it deliberately and update downstream consumers rather than allowing a silent structural change.

How to interpret changes

A different export does not necessarily mean that the underlying data changed. First separate collection changes from content changes. A lower row count could indicate a new filter, a failed page request, a renamed selector, or a genuinely shorter report.

Compare runs in this order:

  1. Collection health: response status, final URL, table presence, and pagination completion.
  2. Schema: header names, column count, data types, and missing-field rates.
  3. Volume: total rows and duplicate identifiers.
  4. Values: added, removed, or modified records.

Use a stable identifier when comparing records. If none exists, create a cautious composite key from fields that should remain consistent, such as a date and reference number. Avoid using row position as an identifier because sorting changes can make every row appear different.

For technical SEO scraping, the same principle applies when tracking tables of URLs, status values, or structured data references. If the page also contains JSON-LD, extract it separately rather than treating it as another visual table; see the guide to extracting JSON-LD and schema markup.

When to revisit

Revisit this workflow whenever the recurring source changes or an export no longer passes its checks. A monthly or quarterly review should confirm that the selector still identifies the intended table, the headers retain their meaning, and the output schema remains useful to the people or systems consuming it.

Update the parser sooner when you observe a redesign, a new pagination method, a move from server-rendered HTML to JavaScript-rendered content, changed character encoding, or a new layout using merged cells. Also revisit the workflow when a downstream spreadsheet, database import, or API rejects the file.

Before putting a revised scraper into a recurring schedule, save a representative HTML fixture and test the parser against it. Compare the new output with a known-good export, review a sample of records manually, and record the schema change. For larger jobs, add retry handling, logging, and validation as described in this web scraping pipeline architecture guide.

The practical checklist is simple: inspect the HTML, map the fields, parse conservatively, preserve links and missing values, validate row structure, export with explicit encoding, and compare each run with the previous checkpoint. That process turns a one-off attempt to scrape an HTML table into a repeatable data extraction workflow that remains useful when the source changes.

Related Topics

#Web Scraping#HTML Tables#CSV#JSON#Data Extraction#Python
S

Scraper Studio

Editorial Team

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.