The Complete Technical SEO Guide

Why technical SEO comes first

Search engines have to discover a page, fetch it, render it and decide what it is about before it can rank. Any failure along that chain makes the rest of your marketing invisible. A brilliant article that sits behind a broken canonical tag, a product page that the crawler never reaches, or a template that takes six seconds to paint on a phone will not rank no matter how good the words are. This guide walks through each stage in the order search engines experience it, so that fixes land in the order that matters. If you would rather have us run the process on your site, start with the Standard SEO & AI Readiness Audit, or the Extended tier for larger or more complex sites.

Technical SEO is also the layer that answer engines depend on most. ChatGPT, Perplexity, Gemini and Google AI Overviews read the same HTML, the same structured data and the same robots directives as Google's crawler, and most of their crawlers do not execute JavaScript. A site that is technically sound for search is most of the way to being readable by AI. A site that is not will be invisible to both.

Crawling

Crawling is how search engines discover URLs. A crawler starts from pages it already knows, follows the links it finds, and schedules the new addresses for fetching. How much of your site gets fetched, and how often, depends on crawl budget, robots directives and the shape of your internal linking. Our article on what crawl budget is and why it matters covers this stage in depth.

Robots and sitemaps

Robots directives tell crawlers what to skip. The robots.txt file blocks paths; the robots meta tag and X-Robots-Tag header control indexing and following on a page-by-page basis. The most common mistake is blocking CSS or JavaScript that the page needs to render, which leaves the crawler with a page it cannot understand. The second most common is a rule left over from a staging site that disallows everything. Check both before anything else.

XML sitemaps tell crawlers what matters and when it changed. A good sitemap lists only canonical, indexable URLs that return a 200 status, carries accurate last-modified dates, and is split into files by section so that problems can be diagnosed per template. A sitemap that includes redirects, noindex pages or 404s teaches the crawler to distrust it.

Server logs

The crawler's behaviour is recorded in your server logs, and the logs are the only source that shows what actually happened rather than what could happen. Four weeks of logs will show which URLs receive the most fetches, how many fetches are wasted on redirects and parameters, and which important pages have not been visited in months. Every serious technical audit starts here, because a crawl tool can only simulate what the log records.

Rendering

Modern sites often build their content in the browser with JavaScript. Search engines can render JavaScript, but they do it later and with a smaller budget, and most AI crawlers do not do it at all. Crawl your site once with rendering off and once with it on. Any content that exists only in the rendered version is at risk. The fix is usually to render the important content on the server, which is what this site does for every page.

Indexation

Indexation is the decision to store a page and make it eligible to appear in results. Being crawled does not guarantee being indexed; search engines drop pages they consider duplicate, thin or unimportant. Search Console's page indexing report shows the split between indexed and excluded URLs and the reason for each exclusion, and that report is the first thing to read when traffic is flat.

Canonical URLs

A canonical tag tells search engines which version of a page is the one to index when several addresses show the same content. Trailing slashes, uppercase letters, tracking parameters, HTTP and HTTPS, and www and non-www hosts all create duplicates. Every page should carry a self-referencing canonical, every duplicate should point at its canonical, and every internal link should use the canonical form. This site enforces all three at build time, because canonical mistakes are the quietest way to lose rankings.

Noindex and duplicate content

Some pages should exist without being indexed: search result pages, thank-you pages, filtered listings, paginated archives beyond a certain depth. A noindex directive keeps them out of the index while allowing their links to be followed. The error to avoid is a noindex that lands on a page that should rank, usually because a template setting was copied from somewhere else. The other error is duplicate content that is not marked as such: a product available under three category paths, or a location page cloned for every suburb with only the suburb name changed.

Site architecture

A clear hierarchy with hubs and spokes distributes authority and makes topical relationships explicit. Search engines infer what a site is about from how it links to itself, and pages close to the home page are treated as more important than pages buried five clicks deep. Every knowledge page on this site belongs to exactly one topic hub for that reason, and every hub links down to its spokes and every spoke links back up.

Internal linking

Internal links are the mechanism that turns architecture into rankings. A page with no internal links pointing at it is an orphan; search engines may never find it, and if they do, they treat it as unimportant. A page linked from the home page and from every related article is signalled as central. Descriptive anchor text matters too: "read more" tells the crawler nothing, while "technical SEO audit for Sydney businesses" tells it exactly what the destination is about. This site fails its own build if any page has no inbound links from another page's body, or if any link uses a banned anchor.

URL structure

URLs should be lowercase, short, stable and readable, without parameters where possible. Once a URL is published it should never change without a permanent redirect, because every change discards the links and history the old address had earned. Flat URLs for articles, with the topic expressed through internal linking rather than through folder depth, allow content to be re-clustered later without breaking anything.

Structured data

Schema.org markup, emitted as JSON-LD, turns page content into facts machines can read: this is an article, written by this person, published on this date, about this organisation, which is located here and offers these services. Search engines use it to understand entities and to power rich results. Answer engines use it to decide whether a page is a trustworthy source and how to attribute it.

What to mark up

Every site needs an Organization node with a name, legal name, address, phone, logo and links to its profiles elsewhere on the web. Every page needs a WebPage node and a breadcrumb. Articles need an author who is a real person with a profile page, a publisher, and dates. Services need a provider and an area served. FAQs on a page should be marked up as FAQPage from the same data that renders on the page, never from a separate source that can drift. Consistency matters more than volume: a site whose organisation is described three different ways is harder for a machine to trust than a site with a single, boring, correct description.

Validation

Markup that is wrong is worse than markup that is missing, because it signals carelessness. Validate every template with the Schema.org validator and Google's Rich Results Test, and check that every identifier referenced inside a page's graph exists. This site does that check on every build.

Performance

Performance is a ranking factor and a conversion factor at the same time. Google measures Core Web Vitals from real users, not lab tools: Largest Contentful Paint for loading, Interaction to Next Paint for responsiveness, and Cumulative Layout Shift for stability. A template that fails any of them on mobile is at a disadvantage against one that passes, and the same slowness costs conversions regardless of rankings.

Where the time goes

Most slow templates share the same causes: images served at the wrong size or in old formats, render-blocking scripts from analytics and chat widgets, web fonts loaded from third-party stylesheets, and layout that shifts as late content arrives. Each has a template-level fix, and template-level fixes repair every page that uses the template at once. Measure with field data, find the largest contributor for each key template, and fix in that order.

Migrations and rebuilds

Rebuilds are where rankings go to die. Before any migration, inventory every ranking URL, every local pack position and every redirect that must survive, and test the new build against that inventory before it launches. Map every old URL to its new home with a permanent redirect, never to the home page. Keep titles, canonicals and structured data at least as good as they were. Then watch the logs and the index coverage report daily for the first month, because a rebuild that lost pages will show up there long before it shows up in revenue.

Measuring the work

Technical SEO is measurable in the order it happens. Logs show whether crawler behaviour changed after a fix. Search Console shows whether indexed page counts moved. Rankings and traffic follow, usually within six to eight weeks for template-level fixes. Baseline every one of those before the first change ships, so that the effect of each release is visible and the next priority is evidence-based rather than guessed.

Definitions, tools and comparisons

The glossary defines the terms this guide uses, starting with crawl budget. The SEO ROI calculator turns a traffic uplift into leads and revenue so a technical fix can be budgeted. If you are choosing a platform to run your own crawls, the Semrush and Ahrefs comparison judges both on the criteria that matter for technical work.

Common questions

The technical SEO FAQs answer the questions we hear most often from clients.