How to Extract Images and Downloadable Files from Websites
Web ScrapingData ExtractionImagesPDFsFile DownloadsAutomation

How to Extract Images and Downloadable Files from Websites

SScraper Studio
2026-11-10
7 min read

A practical checklist for extracting images, PDFs, and downloadable files while resolving URLs, validating types, removing duplicates, and building a manifest.

Extracting images, PDFs, and other downloadable files from a website is more reliable when you treat it as a small data pipeline rather than a simple download task. This guide provides a reusable checklist for discovering file URLs, resolving links, handling lazy-loaded media, identifying file types, preserving filenames, removing duplicates, and exporting a manifest that can be checked or rerun later.

Overview

A website file-extraction workflow usually has five stages: discover candidate URLs, normalize them, validate the responses, download the files, and record what happened. Keeping these stages separate makes the job easier to debug and safer to repeat.

Start by defining the scope. Decide whether you need only files linked from a particular page, every file in a section, or assets loaded after interaction. Also decide which file types count. For example, an image collection may include jpg, png, webp, and svg, while a document archive may include pdf, docx, xlsx, and compressed files.

Before requesting pages, check the website's access instructions and applicable terms. Use a reasonable request rate, identify your client where appropriate, and do not attempt to bypass authentication, access controls, or technical restrictions. If the files require a login, a session-based workflow may be necessary; see Authenticated Web Scraping: Cookies, Sessions, CSRF Tokens, and Login Flows for the related workflow.

A useful output is a manifest rather than a folder full of unexplained files. Each record should retain the source page, original URL, final URL after redirects, local path, detected content type, byte size, download status, and a hash. This gives you an audit trail and lets you identify unchanged or duplicate files during a later run.

Checklist by scenario

Scenario 1: Images are present in ordinary HTML

  1. Fetch the page with an HTTP client and inspect every img element.
  2. Collect src, srcset, and relevant attributes such as data-src or data-original.
  3. Parse srcset into separate URLs instead of treating the whole attribute as one value.
  4. Resolve relative paths against the page URL using URL-join logic.
  5. Remove fragments, trim whitespace, and normalize equivalent URLs where safe.
  6. Choose whether to download every responsive variant or only the largest candidate.

Do not assume that the visible image URL is always in src. A page may place a placeholder there and store the actual asset in a data attribute. Captions, links around the image, and nearby JSON may also contain the original or higher-resolution file.

Scenario 2: Images are lazy-loaded or rendered by JavaScript

  1. Inspect the initial HTML for lazy-loading attributes and embedded state objects.
  2. Check network requests in a browser's developer tools to identify the request that returns the media.
  3. Use a browser automation tool such as Playwright when scrolling or interaction is required.
  4. Wait for the relevant content, not an arbitrary long delay, when possible.
  5. Scroll through galleries or click pagination only within the defined scope.
  6. Capture the rendered DOM and network-discovered URLs, then apply the same normalization and validation steps.

Browser automation is useful for discovery, but it does not always need to handle the final download. A common pattern is to use Playwright to reveal URLs and an HTTP client to download them with controlled retries and streaming. The Playwright web scraping guide and dynamic website workflow cover the browser side of this approach.

Scenario 3: PDFs and files are linked from a page

  1. Collect links from a elements and inspect their href values.
  2. Include links whose path or query suggests a file, but do not rely on the extension alone.
  3. Resolve relative, root-relative, and protocol-relative URLs against the source page.
  4. Follow redirects only as allowed by your client and record the final URL.
  5. Inspect the response headers and initial bytes before assigning a file extension.
  6. Stream large files to disk instead of loading the complete response into memory.

A PDF may be served through a download endpoint with no .pdf suffix, while a URL ending in .pdf may return an access page or an error response. The response's Content-Type, status code, and file signature are more useful than the URL extension by itself. Treat a successful HTTP response as a starting point for validation, not proof that the body is the expected file.

Scenario 4: You need a repeatable archive

  1. Assign each source page a stable identifier.
  2. Store the original URL separately from the normalized and final URLs.
  3. Generate a safe local filename from the server filename, page context, or a deterministic identifier.
  4. Prevent path traversal by stripping directory components and unsafe characters.
  5. Calculate a cryptographic hash after downloading each file.
  6. Write a CSV or JSON manifest, including failures and skipped duplicates.
  7. Use the manifest on subsequent runs to avoid downloading unchanged content unnecessarily.

A practical JSON record can look like this:

{
  "sourcePage": "https://example.com/resources",
  "originalUrl": "/files/guide.pdf",
  "finalUrl": "https://example.com/files/guide.pdf",
  "localPath": "files/guide.pdf",
  "contentType": "application/pdf",
  "bytes": 248193,
  "sha256": "...",
  "status": "downloaded"
}

What to double-check

URL resolution and duplicates

Use the page's canonical base URL when resolving relative links, and account for a page-level base element if one is present. Keep query strings when they may select a specific file or image transformation. For duplicate detection, compare normalized URLs first, then compare content hashes after downloading. Two different URLs can return identical bytes, while one URL can return different content over time.

Headers, redirects, and response content

Some servers vary responses according to headers such as Accept or User-Agent. Send only headers that are appropriate for your workflow and avoid copying browser session headers indiscriminately. Record status codes, redirects, and response content types. If a request returns HTML where an image or PDF was expected, save the error information in the manifest rather than silently writing it as a media file.

Filenames and file types

Prefer a filename supplied by the response's content-disposition header when it is valid, but preserve the source URL and page context as separate metadata. Sanitize names for the target operating system, handle Unicode consistently, and resolve collisions deterministically. For type detection, compare the declared MIME type with the file signature and, where practical, inspect image dimensions or PDF structure.

Pagination and coverage

Confirm that the extraction covers all intended pages, gallery states, and pagination links. Keep a count of discovered URLs per source page and log pages that could not be fetched. If the goal is technical SEO or content inventory work, you may also want to combine the file manifest with page metadata or structured data. The guide to extracting JSON-LD and schema markup can help when file information is exposed in structured data rather than ordinary links.

Common mistakes

  • Scraping only visible HTML: This misses lazy-loaded assets, responsive image candidates, and files revealed by JavaScript.
  • Trusting extensions: A URL ending in .pdf is not enough to establish that the response is a PDF.
  • Saving the URL as the filename: Query strings and long paths produce unstable, unsafe names. Generate a controlled local path.
  • Ignoring redirects: The original link may point to a CDN, signed URL, or download handler. Record both URLs.
  • Loading large files into memory: Stream responses in chunks and set sensible size limits for the job.
  • Skipping hashes: Without hashes, it is difficult to distinguish duplicates or verify whether a later run changed a file.
  • Retrying everything indefinitely: Use bounded retries, backoff, and clear failure states instead of creating an uncontrolled request loop.
  • Mixing discovery and storage logic: Separate URL collection, validation, download, and manifest writing so each stage can be tested independently.

For larger projects, consider a queue with fields for priority, attempt count, status, and last error. If the source is an API rather than rendered HTML, compare the API and HTML approaches before building the extractor; the API versus HTML scraping guide provides a useful decision framework.

When to revisit

Revisit this checklist before a seasonal planning cycle, a content migration, a document-library archive, or any recurring media collection. It should also be reviewed when the source website changes its templates, image delivery, pagination, authentication flow, or download endpoints. Browser automation libraries and HTTP clients change over time as well, so rerun a small test set after updating dependencies.

A practical maintenance routine is:

  1. Keep five to ten representative pages as a regression set, including an ordinary page, a lazy-loaded page, a paginated page, and a page with a failed or protected resource.
  2. Run the extractor against that set after workflow or tool changes.
  3. Compare discovered URL counts, response types, file sizes, and hashes with the previous manifest.
  4. Review failures manually before increasing the crawl scope.
  5. Update filename rules and file-type allowlists when the archive requirements change.

Finally, preserve the manifest with the downloaded files. A folder tells you what exists; a manifest tells you where each item came from, how it was retrieved, and whether it can be verified. That small record is what turns a one-off web scraper into a dependable file-extraction workflow.

Related Topics

#Web Scraping#Data Extraction#Images#PDFs#File Downloads#Automation
S

Scraper Studio

Editorial Team

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.