Requests-HTML - Checking out a new HTML parsing library for Python

Requests-HTML is positioned as a fast, Pythonic way to fetch and parse HTML using the same author as the popular Requests library. The core pitch is simple: pull a page with `requests.get`, then use Requests-HTML’s built-in helpers to inspect encoding, headers, redirects, links, and structured content without reaching for heavier parsing workflows.

After installing with `pip install requests-html` (requiring Python 3.6+), the walkthrough uses a controllable test page (“/parse-me”) to demonstrate the main ergonomics. A single fetch returns an object that supports convenient properties and methods. The HTML can be accessed in multiple forms: “raw” HTML preserves the original formatting, while the library’s formatted HTML normalizes whitespace and indentation. Link extraction is equally straightforward—calling the links-related helper returns a set of URLs, deduplicating repeats. The object also exposes metadata like response encoding, cookies, whether a redirect occurred, and the ability to convert responses into JSON when the server provides JSON.

The most distinctive capability is JavaScript rendering. Instead of treating the page as static HTML, Requests-HTML can run client-side JavaScript via a headless browser. The workflow is to fetch the page, then call `.render()` to execute scripts and update the DOM before querying it. In the example, the rendered output includes text that only appears after JavaScript runs, confirming that the library can extract post-JS content.

On Windows, however, the rendering step hits a practical edge: Chromium is downloaded and rendering succeeds, but cleanup fails when the library tries to delete its temporary user-data directory. The error is traced to a permissions issue during directory removal (using `shutil.rmtree`), and the presenter experiments with toggling error-ignoring behavior. Even with elevated privileges, deletion still fails, so a workaround is to avoid raising the exception and continue, or manually delete the temporary directory later.

Beyond basic parsing, the tutorial experiments with searching the DOM—using methods like `.find()` to locate elements such as links, IDs, divs, and table cells. The search syntax is shown to be quick, but also somewhat finicky, especially when targeting attributes or classes with spaces and when mixing selector-like patterns.

Finally, the walkthrough tests Requests-HTML’s pagination support, which can follow “next page” links automatically through an iterator-like interface. It appears to work on Reddit, but the pagination behavior is inconsistent on other sites. On Hacker News, the “next” traversal sometimes lands on the wrong page, suggesting that pagination detection is heuristic and may require tuning or improvement.

Overall, Requests-HTML is presented as a strong alternative to BeautifulSoup for straightforward HTML parsing and for cases where JavaScript-rendered content matters. The main caveats are platform-specific cleanup issues during rendering (notably on Windows) and imperfect pagination across different site structures.