Remigiusz ZalewskiRemigiusz Zalewski

Master Web Scraping in .NET: Extract Data from Any Website

web-scraping-in-dotnet
Sponsor this newsletter →

Introduction

An API is the polite way to get data out of a website; scraping is what's left when there isn't one. This video builds a scraper in .NET using HttpClient to fetch the raw HTML and HtmlAgilityPack to actually navigate it - plus the point where a plain HTTP request isn't enough because the data you want doesn't exist until JavaScript runs in a real browser.

🎬 Watch the full video here:

Fetching the raw HTML

var html = await httpClient.GetStringAsync("https://example.com/products");

This is the entire "fetch" step for a static page - no browser involved, just an HTTP GET returning whatever HTML the server sent, before any client-side JavaScript would have had a chance to modify it.

Parsing with HtmlAgilityPack

Raw HTML as a string is unusable for extracting structured data reliably - HtmlAgilityPack parses it into a navigable DOM, tolerant of the malformed, not-quite-valid HTML real websites actually ship:

var doc = new HtmlDocument();
doc.LoadHtml(html);

var titles = doc.DocumentNode
    .SelectNodes("//div[@class='product-card']//h3")
    .Select(node => node.InnerText.Trim());

SelectNodes takes an XPath expression - //div[@class='product-card']//h3 reads as "any h3 nested anywhere inside a div with class product-card, anywhere in the document." XPath is more verbose than a CSS selector but ships in HtmlAgilityPack without an extra package; HtmlAgilityPack.CssSelectors adds QuerySelectorAll for people who'd rather write .product-card h3.

Extracting attributes, not just text

var links = doc.DocumentNode
    .SelectNodes("//a[@class='product-link']")
    .Select(node => node.GetAttributeValue("href", string.Empty));

GetAttributeValue with a default value avoids a null reference on a node that's missing the attribute entirely - worth defaulting defensively, since scraped HTML is never as consistent as data from your own API.

When a plain HttpClient GET isn't enough

The moment a page renders its actual content via client-side JavaScript after load - a React or Angular single-page app, infinite-scroll content loaded on demand - GetStringAsync returns the initial HTML shell, not the populated page a user actually sees in a browser. That requires a real (or headless) browser engine like Playwright or Selenium to execute the JavaScript first and then extract from the resulting DOM - a fundamentally different, heavier tool than an HTTP GET plus a parser.

Scraping responsibly

Before scraping anything, check the site's robots.txt and terms of service - some explicitly disallow automated scraping, and rate-limiting your own requests (rather than hammering a site as fast as the network allows) is both more polite and less likely to get your IP blocked.

Common pitfalls

  • Assuming GetStringAsync sees what a browser shows. It doesn't run JavaScript - client-rendered content needs a headless browser instead.
  • Brittle selectors tied to a class name a site can rename at any time. Scraping code inherently breaks when the target site's markup changes; expect and plan for maintenance.
  • No rate limiting. Hammering a site with rapid requests risks getting blocked and is inconsiderate regardless.
  • Not checking robots.txt or terms of service before scraping a site at all.
  • Not handling missing nodes/attributes defensively. Real-world HTML is inconsistent; a SelectNodes call can return null for a page that doesn't match the expected structure.

Key Takeaways

  • HttpClient.GetStringAsync fetches raw, pre-JavaScript HTML - HtmlAgilityPack then parses it into a navigable, malformed-HTML-tolerant DOM.
  • XPath expressions (or CSS selectors via an extra package) locate the elements you actually want to extract.
  • Client-rendered pages need a headless browser (Playwright, Selenium) instead of a plain HTTP GET, since the content doesn't exist until JavaScript runs.
  • Check robots.txt and rate-limit requests - scraping without either is both impolite and a good way to get blocked.

Get the Full Source Code

The complete runnable solution - the HttpClient + HtmlAgilityPack scraper with XPath selectors - is available to Patreon supporters. If you want the extraction logic ready to point at your own target site instead of rebuilding it from the walkthrough above, you can find it on Patreon.

Resources