Turning an HTML table into dependable CSV or JSON takes more than selecting rows and joining cells. This practical checklist shows how to detect headers, preserve links and data types, handle merged cells and multiple tables, and validate the result with reusable Python examples and export rules.
Overview
An HTML table can look structured in a browser while remaining inconsistent in the source. Header cells may appear in a <thead>, inside the first row, or across several rows. A cell may contain a link, an image, a line break, or a nested list rather than plain text. Some tables also use rowspan and colspan to represent values visually without repeating them in every row.
A reliable table scraper therefore has four stages: identify the correct table, map its structure to a schema, normalize each cell, and validate the export. Treat CSV and JSON as different outputs rather than interchangeable formats. CSV is convenient for spreadsheets and simple tabular imports, but it represents every value as a field in a row. JSON can preserve nested links, source attributes, and metadata more naturally.
Before extracting data, check whether the site offers a documented API or downloadable file. If HTML is the appropriate source, review the site’s terms, access requirements, and technical constraints. For JavaScript-rendered pages, a browser automation tool such as Playwright or Puppeteer may be needed to obtain the table after the page has loaded. For server-rendered HTML, a direct HTTP request and an HTML parser are usually simpler and easier to operate.
Keep a raw copy of the response and record the source URL, retrieval time, selected table, and parser version. This makes it possible to explain changes later instead of treating every unexpected row count as a data mystery.
Checklist by scenario
1. One simple table with one header row
For a conventional table, start by selecting the intended <table>, locating header cells, and extracting one value per body row. The following Python example uses Requests and Beautiful Soup:
import csv
import requests
from bs4 import BeautifulSoup
url = "https://example.com/report"
html = requests.get(url, timeout=30).text
soup = BeautifulSoup(html, "html.parser")
table = soup.select_one("table.results")
headers = [cell.get_text(" ", strip=True) for cell in table.select("thead th")]
rows = []
for tr in table.select("tbody tr"):
cells = tr.find_all(["th", "td"])
values = [cell.get_text(" ", strip=True) for cell in cells]
if values:
rows.append(dict(zip(headers, values)))
with open("report.csv", "w", newline="", encoding="utf-8-sig") as output:
writer = csv.DictWriter(output, fieldnames=headers)
writer.writeheader()
writer.writerows(rows)Use a specific selector when possible. Selecting the first table on a page is fragile because navigation, comparison widgets, and footer elements may also contain tables. If the table has no <thead>, inspect the first row and use its <th> cells as headers, but do not assume the first row is a header without checking the markup.
2. A table with nested links or useful attributes
Plain text extraction loses information when a cell contains a link. Decide whether the export needs the visible label, the destination URL, or both. A JSON record can preserve both:
def extract_cell(cell):
link = cell.find("a", href=True)
value = cell.get_text(" ", strip=True)
return {
"text": value,
"href": link["href"] if link else None
}
records = []
for tr in table.select("tbody tr"):
cells = tr.find_all(["th", "td"])
if len(cells) == len(headers):
records.append({
header: extract_cell(cell)
for header, cell in zip(headers, cells)
})For CSV, flatten the structure into explicit columns such as product_name and product_url. Resolve relative URLs against the page URL before exporting if downstream users need clickable, complete links. The same approach applies to image sources, data attributes, and download URLs; see the guide on extracting images and downloadable files for related handling.
3. Merged cells using rowspan or colspan
Merged cells require a grid-building step. A value with rowspan should be carried into the following rows, while a value with colspan occupies more than one column. A parser that simply loops over cells will produce shifted fields and misleading records.
Build an intermediate two-dimensional grid before applying column names. Track occupied positions, place each cell in the next available column, and fill its declared span. Decide explicitly whether repeated values are desired in the final data. For analysis-ready output, repeating a grouped label in each affected row is usually easier to filter than leaving blank cells. Preserve the original merged structure separately if visual fidelity matters.
4. Multiple tables on one page
When a page contains several tables, create a selection rule based on a class, caption, nearby heading, column signature, or a combination of these. Compare the detected headers against the expected schema before extracting. If a report contains monthly tables with the same columns, add a source-period field to every record rather than relying on the file name alone.
If tables have different schemas, export them as separate JSON arrays or separate CSV files. Forcing unrelated tables into one wide CSV often creates empty columns and hides the distinction between record types.
5. JavaScript-rendered tables
If the initial HTML contains no rows, inspect the network requests and page behavior before switching tools. The table may be populated by an API call that is a better data source, subject to access and usage requirements. If browser rendering is necessary, wait for a stable selector such as the table body or a known row count, then capture the rendered DOM. Avoid using a fixed sleep as the only readiness check because loading time can vary.
What to double-check
- Headers: Confirm their order, uniqueness, and meaning. Rename blank or duplicated headers with stable field names.
- Row counts: Compare the number of source rows with the number of exported records. Explain intentionally skipped rows, such as separator rows or pagination controls.
- Types: Preserve identifiers as strings when leading zeroes matter. Convert numbers only after removing known formatting such as thousands separators. Keep dates in a consistent, documented format.
- Whitespace and encoding: Collapse presentation-only whitespace while preserving meaningful line breaks when required. Open the export using the expected UTF-8 encoding and test non-ASCII names or symbols.
- Missing values: Choose one convention, such as an empty string or JSON null, and apply it consistently. Do not confuse a missing value with a genuine zero.
- Duplicates: Check whether repeated rows represent duplicate source content, pagination overlap, or legitimate repeated records. Normalize and deduplicate only with a defined key; the guide on cleaning and deduplicating scraped data provides a broader workflow.
- Pagination: Determine whether the visible table is one page of results. Record page numbers or follow pagination deliberately, then validate the combined total.
- Traceability: Keep the source URL and retrieval timestamp in a sidecar file or JSON metadata object. This is especially useful for recurring technical SEO or reporting workflows.
A compact quality-control rule is: inspect a sample from the first, middle, and last part of the export; compare expected and actual columns; count records; test a row containing a link or unusual character; and confirm that numeric and date fields can be read by the next system.
Common mistakes
Choosing the first table: Page layout changes can make an unrelated table become the first match. Select by meaningful structure and fail clearly when no match is found.
Using visible text as the only value: This discards URLs, identifiers, and attributes. Extract the fields that downstream users actually need.
Assuming every row has the same number of cells: Group headers, notes, merged cells, and expandable rows can violate that assumption. Filter known non-data rows and normalize the grid before mapping values.
Converting everything to numbers: Product codes, postal codes, and account identifiers may look numeric but are not quantities. Define types by meaning, not appearance.
Ignoring pagination or lazy loading: A successful request does not prove that all records were collected. Check for next-page controls, total counts, and API calls.
Overwriting raw input: Clean exports are easier to debug when the original HTML is retained. Store raw and processed data separately, with clear filenames and timestamps.
Running a parser without safeguards: Add timeouts, selector checks, row-count thresholds, logging, and a failure path. For scheduled jobs, follow a repeatable deployment and monitoring process such as the one described in reliable scraping job workflows.
When to revisit
Revisit your HTML table scraper before a seasonal planning cycle, whenever the source report changes, or when a workflow moves from manual inspection to scheduled collection. A redesign may change selectors, headers, pagination, encoding, or rendering behavior without changing the table’s appearance to a casual visitor.
Set a lightweight review trigger around each run: alert when the expected table is missing, the column set changes, the row count falls outside a reasonable range, or a high proportion of values becomes empty. Do not treat every change as an error; classify it as an expected schema update, a temporary source problem, or a parser defect.
Before putting the scraper back into production, rerun the checklist: identify the table, confirm the schema, test merged and nested content, verify types and encoding, compare row counts, and inspect representative records. If the task expands beyond tables into structured metadata, pair it with the guide on extracting JSON-LD, Open Graph, and microdata. If the table is only one part of a larger crawl, review the approach to reliable data-extraction pipeline architecture.
The goal is not merely to produce a file that opens in a spreadsheet. It is to create an export whose structure, provenance, and quality can be checked the next time the source changes.