Using XPath for Better, More Meaningful Crawls

When I compare what SEO tools offer on the crawling side, almost all of them ship the same features.

Slow pages, duplicate tags, title/description/H1 problems, link depth, redirected URLs, and so on. I could keep going.

Any crawler used with its default settings tends to produce results that are not particularly effective.

And if we assume even a low-authority competitor is running the same kind of audit, then we need more data, more analysis and more meaningful conclusions.

That is why I want to talk about using XPath to make crawls far more effective.

How to get more meaningful data with Screaming Frog and XPath

Some examples:

1 — I can collect the reading time of every article published on this blog.

If you only want the how-to, you can skip these examples and jump to the XPath section.

2 — The product pages on Bitturk.com carry a generous FAQ section. While crawling the site we can use XPath to collect every FAQ question in one pass.

3 — Say we are about to build a landing page for a product.

Which questions do people ask most about it? What are they happiest with? What do they complain about? To answer questions like these and improve the on-page experience, we can harvest the product reviews.

For the "Samsung Galaxy Note FE" we can pull every review from Hepsiburada.

To make sense of those reviews we can filter the most repeated words in Excel and interpret them.

We could multiply the examples, but what matters is deciding which data you actually need — and being able to interpret it once you have it.

By identifying the shared patterns across many pages and defining them in the tool before the crawl, we can grow the data set the crawl produces.

Let's get to using XPath

First, look at the source of the data you want to collect. It could be the product price, the number of reviews, the reviews themselves. It depends on the need.

I want every review for the product.

I can inspect the page holding the reviews with the Chrome console.

Looking at the review area, its class is "review-text". We can use the Chrome console to grab the XPath of that class — but the console does not always hand you the right value.

So you may need to build the XPath by hand (in the form //*[@class="review-text"]).

Or, as an alternative, you can use the Scraper extension for Chrome.

Collecting data with XPath in Screaming Frog

Before starting the crawl, the XPath entries have to be filled in correctly under Config > Custom > Extraction.

Because the reviews I am collecting only apply to one product, I will crawl nothing but the pages holding those reviews. If the number of pages to crawl is large, I generate the URLs in Excel and then let Screaming Frog crawl only that list.

Happy crawling.