Mapping Site Architecture with Screaming Frog and Gephi

The internal link graph matters a great deal in technical SEO. When Google scores pages it pays attention to internal links as well as external ones. If a page receives links from many different pages within the site, it encourages Google to crawl that page more often and passes value to it.

Since every page carries a different value, internal link optimisation has to be done deliberately for the pages that matter.

Large sites can be hard to get to know: which categories are strong, which pages sit under which category, where the traffic is concentrated, which page targets promising keywords but gets no traffic. Questions like these are how you learn a site.

Wouldn't it be nice to pour the structure into a picture — a tree — to see it up close?

Internal linking can be driven from the header menu, the footer, in-page text, the sidebar or a page-specific menu.

But which page is linked from which? How many links does it get? Which page receives the most?

Why do site structure and internal links matter?

Making the site easier to crawl: as Googlebot moves through your site it takes the links on each page into account and wants to crawl them. We do not want an important page to be crawled rarely — or not at all.

Letting value flow: alongside many other algorithms, Google uses PageRank to determine the value of a page. Every page on the site shares value with the others through links. Getting the right links and the right structure is therefore what carries value to the pages that matter.

Improving user experience: like Googlebot, users want to reach what they are looking for easily. The site menu and the other links on a page should guide them. Pages that were forgotten in the internal linking, or that sit too deep, are hard for users to reach.

So let's begin.

After crawling the site with Screaming Frog, download every link each page receives from Bulk Export > All Outlinks. The exported sheet needs a small clean-up: column 1 should hold "Source" values and column 2 "Target" values. (You can delete everything else.)

Now open Gephi. (You can download it free from its own site.)

Create a new empty project and import the sheet: File > Import Spreadsheet.

Once the file is imported the table loads and the first network graph appears.

Because the site is large there are a great many outgoing links, so the first picture is meaningless. To make the graph and the structure readable you need the program's own features: colour things from the Appearance panel, apply filters, and you will get far more meaningful visuals.

A few resources for using Gephi more effectively:

Easy Visualizations of PageRank by Page Groups with Gephi (Search Engine Land)

Plotting a Website's Structure with Gephi (Onely)

Visualizing Internal Link Juice Flow and Power with Gephi (AWR)

Visualize your Website Structure with Gephi (Analysith)

Internal link mapping (Kazu)

Visualizing Site Structure for SEO Analysis (LinkedIn)