EXTERNAL-Universal-Sitemap-Visualiser-KB

Sitemap Inspector

A browser-based toolkit for exploring a large website’s information architecture from a Slickplan sitemap export, without scrolling through a flat tree. Upload a CSV, get eight different views of the same hierarchy, and switch between them with a click. Nothing is uploaded anywhere — the CSV is parsed entirely in your browser.

Live use: open index.html via a local server (see Running it below) or host the folder on GitHub Pages. It loads a bundled sample sitemap (data/sample-sitemap.csv) on first load; use upload CSV in the top bar to load your own export.


Why this exists

A flat indented tree (what Slickplan and most sitemap tools show by default) works fine for a hundred pages. It stops working once a site has thousands — you spend more time scrolling and collapsing than thinking. This tool exists to answer three specific questions that a flat tree answers badly at scale:

What it accepts as input

Upload a file (upload file button) — three formats, auto-detected by extension and content:

Load from a URL (load URL button) — point it at a live sitemap, e.g. https://www.port.ac.uk/sitemap.xml. It handles the two ways big sites split their sitemaps:

Everything is merged, deduplicated, and capped at 25,000 URLs to keep the browser responsive. Progress shows in the readout bar while fetching.

A note on CORS: browsers block cross-origin reads by default, and most sites don’t allow them on their sitemap. The tool tries a direct fetch first, then automatically retries via a public CORS relay (allorigins.win, then corsproxy.io). That works for nearly everything, but it does mean the sitemap URL transits a third-party service when the direct fetch fails — fine for public sitemap data, but worth knowing. If a site blocks both routes, download the sitemap.xml manually and use upload file instead.


The eight views

01 — Outline

The plain indented list, but collapsible and searchable. This is the view every other visualisation is checked against — if a different view shows something surprising, come back here to confirm what’s actually in the data. Click a row’s name to inspect it in the readout bar; click the ▾/▸ to fold a branch.

02 — Tree

A node-link diagram (the shape people usually mean by “site tree”). To stay usable at scale, every top-level section starts collapsed — you see all ~100+ sections as a single screen, and drill into one by clicking it. Search expands the path to any match automatically. There’s no virtual scrolling limit here, so very deep branches (this sample goes 7 levels deep) just keep extending rightward; the view scrolls.

03 — Sunburst

The same hierarchy as a set of concentric rings — depth = distance from the centre. This is the fastest way to see where the structure is unusually deep (a thin ring reaching far out from a small section) versus where it’s unusually broad (a section with many same-depth children, shown as many slices in one ring). Click a slice to zoom into that branch; click the centre circle to zoom back out.

04 — Treemap

Tile area = number of pages in that branch. This is the single best “whole site on one screen” view — bloated sections are obvious immediately because they’re visibly bigger tiles, with no scrolling required even at 1,000+ pages. Shows two levels at a time (a section and its immediate children); click a tile to drill into it, use the breadcrumb at the top to climb back out.

05 — Circle pack

The same nesting as the Treemap, but as packed circles instead of rectangles. Some people read circular nesting more intuitively than rectangular tiling, especially for spotting “one branch is much bigger than its siblings.” Click a circle to zoom in, click empty space to zoom out.

06 — Themes

This is the odd one out, and the one built specifically for IA-redesign conversations. It throws away the current navigation structure and regroups every page by a keyword detected in its name/URL slug (see How theme detection works below). The point isn’t that this clustering is “more correct” than the real IA — it’s a second opinion you can compare against the first five views. Hover a page here, then check the readout bar: it always shows that page’s real, current breadcrumb. If a topic’s pages are scattered across several theme bubbles while sitting together in one IA branch (or the reverse — one theme bubble pulling together pages that are currently scattered across five different sections) that’s worth a conversation.

07 — 3D graph

A force-directed network in navigable 3D space (drag to orbit, scroll to zoom, click a node to fly to it). Functionally similar information to the Tree view, but nodes are free to spread out in 3D rather than being constrained to a fixed layout, which can make it easier to get an intuitive sense of a large, messy graph at a glance. This is the most exploratory of the eight — it’s genuinely fun to fly around, but for precise structural reading the 2D views are more reliable.

08 — 3D city

Each top-level section is a ground plot; each of its child pages is a tower, height = how many pages sit beneath that child. Plot colour reflects the section’s own dominant theme; tower colour reflects the individual page’s theme, or the flag colour if it’s been flagged. This is the most stakeholder-friendly view in the set — “this district is clearly overbuilt” reads instantly to someone who doesn’t normally look at sitemaps. It’s illustrative, not exhaustive: very large sections show their 24 biggest children with a “+N more” marker rather than every descendant, since rendering thousands of individual towers per district would be unreadable and slow. Use the Treemap or Outline for the exact count.

09 — Walkthrough

A first-person corridor walk through the IA. Each level of the site is a hallway; child pages are doors set ajar along the walls, with the page name painted on the door face itself (plus its sub-page count, or “leaf page”). Scroll or ↑/↓ (or W/S) to walk, click a door to enter that section — the corridor rebuilds for its children — and press Esc or use the back button to go up a level. Doors are coloured by theme (or flag colour), so a corridor full of orange doors is itself a finding. Like the 3D city, it’s illustrative rather than exhaustive: corridors show the first 24 pages of a section with a “+N more” sign at the end, and it’s the most memorable way to give stakeholders a feel for how deep and cluttered a branch really is — walking four corridors down to reach a page makes “our IA is too deep” visceral in a way a diagram doesn’t.


How theme detection works

Deliberately simple and auditable, since this view exists to support a discussion, not to replace it:

  1. Every page’s name and URL slug is tokenised into words, common stopwords removed (the, of, your, page, home, 2025…).
  2. The 20 most frequent remaining words across the whole site become “theme seeds” (a word has to appear at least 3 times to qualify).
  3. Each page is assigned the highest-ranked seed word found in its own name/slug. If none matches, it inherits the seed found in its nearest ancestor’s name/slug. If still nothing matches, it’s labelled other.

This means themes are literally just “common words in your page names,” which is intentional — it’s transparent and a domain expert can immediately sanity-check it, unlike a black-box clustering model. It also means the other bucket can be large on sites with very generic or inconsistent naming; if that bucket feels too big, it’s a sign the page names themselves carry too little topical information, which is itself a useful finding.

How flagging works

Every view can highlight pages worth a second look (toggle with flag check in the top bar). A page is flagged if any of these are true:

Flags are additive signals, not a verdict — a flagged page might be entirely intentional (e.g. a deliberately reused thank-you template). The point is surfacing it for a human to decide, not auto-removing it.


Running it

Open index.html directly from disk and the sample data won’t load — browsers block fetch() of local files over file:// for security reasons. Two easy options:

# from inside this folder
python3 -m http.server 8000
# then open http://localhost:8000 in your browser

or push the folder to a GitHub repository and enable Settings → Pages on it — GitHub Pages serves everything over https://, so fetch works with no further setup. Uploading your own CSV via the upload CSV button works either way, since that reads the file directly rather than fetching it.

About the libraries this depends on

PapaParse, D3, three.js, OrbitControls, and 3d-force-graph are vendored directly in js/vendor/ rather than loaded from a CDN. This was a deliberate fix: an earlier version loaded them from jsdelivr, and on at least one network it consistently failed for one specific resource (3d-force-graph) for reasons that turned out to be a loading-order bug rather than the CDN itself — that library expects a global THREE to already exist when it runs, and a parallel-loading fallback I’d added could race ahead of it. Vendoring everything removes that whole class of problem: no CDN, no race condition, no dependency on the visitor’s network reaching a third party at all. The trade-off is the repo is about 1.6MB heavier and you’ll need to manually re-download a file if you ever want to bump a library version — for a small internal tool, that’s the right trade.

If you do want to update one later: download the package from npmjs.com, grab the relevant .min.js from its dist/ (or build/) folder, and drop it into js/vendor/ under the same filename. Keep three.min.js loading before OrbitControls.js and 3d-force-graph.min.js in index.html — both of those expect window.THREE to already exist.

What’s deliberately not included (yet)

A few visualisation styles came up in scoping and were left out of this first pass because they’re close variants of something already here, with limited extra analytical value for the effort:

Happy to add any of these if they turn out to matter once you’re actually using the other eight.

Known limitations of this first pass