Technical SEO in 2026: the complete audit & optimization guide
By David Sessford · 26 June 2026 · 16 min read
Let's cut the crap. Here is exactly what works.
Technical SEO is the work that lets search engines and AI assistants crawl, render, understand and trust your site. In 2026 that means three jobs: be reachable (crawling and indexing), be fast and stable (Core Web Vitals), and be legible to machines (clean HTML, schema and sensible AI-crawler rules).
Most "technical SEO checklists" are a wall of 150 tick-boxes with no sense of what actually moves the needle. This one is different. It's ordered by impact, it's current to mid-2026. Including Google's December 2025 clarification on how JavaScript error pages get rendered. And every threshold and date is sourced at the bottom. Work top to bottom and you'll fix the things that quietly cap a site's rankings before you ever touch content or links.
Key takeaways
- Crawling and indexing come first. If Google can't reach or won't index a page, nothing else you do to it matters. Fix this layer before speed or schema.
- Core Web Vitals thresholds are fixed and public: LCP under 2.5s, INP under 200ms, CLS under 0.1. And you need 75% of real visits in the "good" band to pass.
- INP replaced FID on 12 March 2024. If your dashboards still talk about First Input Delay, they're out of date.
- Error pages are a rendering trap. Since December 2025 Google says pages returning a non-200 status (like a JS-built 404) may be skipped for rendering entirely.
- AI crawlers are now part of technical SEO. GPTBot, ClaudeBot, PerplexityBot and Google-Extended each behave differently. And you control them in robots.txt.
What technical SEO actually is
Technical SEO is everything that affects how search engines and answer engines access and process your website, as opposed to the words on the page (on-page SEO) or the links pointing at it (off-page SEO). It's the plumbing. When it works, nobody notices. When it breaks, your best content can sit invisible for months.
Think of it as a pipeline a search engine runs every time it meets your site: crawl the URL, render the page (run its JavaScript, see the finished result), index the content if it's worth storing, and rank it against everything else. Technical SEO is making sure nothing falls out of that pipeline. A page that can't be crawled is never rendered. A page that's rendered wrong is indexed wrong. A page that loads like treacle gets a quieter ranking. Each stage gates the next. Which is exactly why this guide is ordered the way it is.
If any of the jargon trips you up as you read, our plain-English SEO & AI-search glossary defines every term used here.
Why it matters more in 2026, not less
You'd think that after twenty years the technical layer would be a solved problem. The opposite is true. The surface a "page" has to satisfy keeps multiplying. In 2026 the same HTML has to be readable by Googlebot, by Bingbot (which also feeds Microsoft Copilot), and by a growing fleet of AI crawlers feeding ChatGPT, Gemini, Perplexity and Claude. They don't all behave the same way, and several of them are far less forgiving of messy, JavaScript-dependent markup than modern Google is.
At the same time, the cost of a technical mistake has gone up. When a slice of clicks is being absorbed by AI Overviews and chat answers, the traffic that does reach classic results is more valuable and more contested. A site that crawls cleanly, loads fast and exposes its facts in plain HTML wins on every surface at once. A site that hides its content behind client-side rendering loses on all of them. Technical SEO has quietly become the shared foundation under both traditional SEO and answer-engine optimisation.
The 2026 technical SEO audit, in priority order
Here's the whole audit as one map. Start at the top: a problem in an upper row usually makes the rows below it irrelevant until it's fixed. Use the "fix-first?" column to triage a site you've never seen before.
| Layer | What you're checking | Typical killer issue | Fix first? |
|---|---|---|---|
| Crawlability | robots.txt, crawl traps, server errors, sitemap | A stray Disallow: / or 5xx errors | Critical |
| Indexation | noindex tags, canonicals, Search Console coverage | Pages stuck "Crawled. Currently not indexed" | Critical |
| Rendering | JavaScript dependency, non-200 pages, content in HTML | Content only appears after client-side JS | Critical |
| Site architecture | internal links, depth, orphan pages | Key pages 5+ clicks from the homepage | High |
| Core Web Vitals | LCP, INP, CLS on real users (CrUX) | Slow LCP from unoptimised hero images | High |
| Structured data | schema validity, eligible rich-result types | Invalid JSON-LD, or none at all | Medium |
| Duplication | canonicals, parameters, hreflang | Faceted URLs creating thousands of dupes | Medium |
| AI legibility | robots.txt for AI bots, clean HTML, llms.txt | Blocking retrieval bots you actually want | Medium |
The rest of this guide walks each layer in that order. You don't need every tool under the sun to do this. Google Search Console plus one crawler covers the vast majority of it, and we'll name the others later.
Layer 1. Crawlability: can a bot even reach the page?
Crawling is a search engine fetching your URLs. If it can't fetch a page, that page cannot rank. Full stop. The single most damaging technical SEO bug in existence is a robots.txt file that accidentally disallows the whole site, usually left over from a staging environment. It happens more than you'd believe. Check yours first, every time.
The crawlability checklist
- robots.txt. Confirm it isn't blocking anything important. Remember robots.txt controls crawling, not indexing: a disallowed page can still appear in results without a snippet if it's linked elsewhere.
- Server health. Recurring 5xx errors tell Google to back off and crawl less. Watch the Crawl Stats report in Search Console for spikes.
- XML sitemap. List your canonical, indexable URLs only; keep it under 50,000 URLs per file and reference it in robots.txt. Don't list redirected, noindexed or 404 URLs in it.
- Crawl budget. For large sites (hundreds of thousands of URLs), stop wasting crawl on infinite calendars, faceted-filter combinations and session-ID URLs. For sites under a few thousand pages, crawl budget is rarely your problem.
- Internal redirects. Chains and loops burn crawl and dilute signals. Point links at the final 200 URL directly.
Layer 2. Indexation: will Google keep the page?
A crawled page isn't an indexed page. Google decides whether each URL is worth storing, and in 2026 it is pickier than it used to be. Thin, duplicative or low-value pages increasingly sit in the "Crawled. Currently not indexed" bucket in Search Console. That status is Google telling you the page isn't earning its place.
The two technical levers here are the noindex robots meta tag and the rel="canonical" link. Misuse either and you can erase yourself from search. A few rules that prevent the most common disasters:
- Never ship a sitewide
noindex. It's the second-most-common catastrophic launch bug after a blocking robots.txt. Audit for stray noindex tags after every deploy. - Don't put
noindexin JavaScript-injected markup if you can avoid it. And definitely don't combine noindex with a robots.txt disallow on the same URL, because if Google can't crawl the page it can't see the noindex either. - Canonicals are a hint, not a command. Make them self-referential on canonical pages and consistent. Don't canonicalise page B to A while internally linking and sitemapping B as if it's the main version. Mixed signals get ignored.
- Use Search Console's URL Inspection to see the indexed version of any page and the canonical Google actually chose, which isn't always the one you declared.
If Google can't reach a page, it can't rank it. If it won't index a page, it can't rank it. Fix the plumbing before you polish the content.
Layer 3. The rendering trap (and the December 2025 change)
This is the layer that catches modern, JavaScript-heavy sites. And the rules tightened at the end of 2025. Google crawls your raw HTML first, then queues the page for rendering, where it runs the JavaScript and sees the finished DOM. Anything that only appears after that JavaScript runs. Your main copy, internal links, even your structured data. Depends on rendering happening at all.
On 18 December 2025, Google updated its JavaScript SEO documentation to make something explicit: every page returning a 200 status code goes to the rendering queue, but a page returning a non-200 status (for example a 404 error page) may have rendering skipped entirely. If your single-page app serves "soft" error pages that return a 200 but show error content, or builds its 404 content with client-side routing, this matters a lot. When Google skips rendering, it never executes your JavaScript, so any content, links or schema that depend on it are simply never seen. And that page will likely rank poorly or not at all.
How to stay out of the trap
- Put the content that matters in the server-rendered HTML. Server-side rendering (SSR), static generation or hydration-friendly frameworks let bots see your copy without waiting on the render queue. AI crawlers especially benefit. Several of them execute little or no JavaScript.
- Make real errors return real status codes. A page that doesn't exist should return 404 or 410, not a 200 with "page not found" baked in by JavaScript. And don't rely on JS to inject critical content onto pages that legitimately return non-200 codes.
- Test with the live tools, not your eyeballs. Use Search Console's URL Inspection "view rendered HTML" and the Rich Results Test to see exactly what Google sees after rendering. If your content is missing there, it's missing from the index.
- Don't block your JS and CSS in robots.txt. Googlebot needs them to render the page the way a user sees it.
Layer 4. Site architecture and internal linking
Architecture is how your URLs connect. It does two jobs at once: it helps crawlers discover pages, and it distributes authority (PageRank) around your site through internal links. A flat, logical structure where important pages sit a couple of clicks from the homepage will out-perform a deep, tangled one almost every time.
- Keep important pages shallow. Aim for money pages within about three clicks of the homepage. Pages buried five or six clicks deep get crawled less and rank worse.
- Kill orphan pages. A page with no internal links pointing at it is hard for Google to find and signals low importance. Every page you care about should be linked from somewhere relevant.
- Use descriptive anchor text. Internal anchors are a ranking signal you fully control. "best link-building tools" beats "click here" every time.
- Build topic clusters. A pillar page linking out to focused supporting articles (and back) concentrates relevance. It's exactly how we structure the guides on this blog, and a fast route to moving page-2 keywords onto page 1.
Internal linking is the cheapest, most under-used lever in technical SEO. You own every link; no outreach required. Done well it lifts whole sections of a site at once. And it's a core part of the system we teach inside The Dojo.
Layer 5. Core Web Vitals in 2026
Core Web Vitals are Google's three public, fixed-threshold metrics for real-user experience. They're a confirmed (if modest) ranking factor, and more importantly they're a direct proxy for the conversion-killing friction your visitors actually feel. Here are the exact 2026 thresholds. These numbers haven't moved, so anyone quoting different ones is guessing.
| Metric | Measures | Good | Needs work | Poor |
|---|---|---|---|---|
| LCP. Largest Contentful Paint | Loading speed | ≤ 2.5s | 2.5-4.0s | > 4.0s |
| INP. Interaction to Next Paint | Responsiveness | ≤ 200ms | 200-500ms | > 500ms |
| CLS. Cumulative Layout Shift | Visual stability | ≤ 0.1 | 0.1-0.25 | > 0.25 |
The catch most people miss: to "pass," at least 75% of real visits to a URL must hit the "good" band, measured on field data in the Chrome User Experience Report (CrUX). Not a one-off lab score from your laptop on fast broadband. Lab tools like Lighthouse are for diagnosis; CrUX is the scoreboard.
The metric that trips people up: INP
On 12 March 2024, Interaction to Next Paint (INP) replaced First Input Delay (FID) as the responsiveness Core Web Vital. FID only measured the delay on a user's first interaction; INP measures responsiveness across the whole visit, capturing the lag from a tap or click all the way to the next frame painted on screen. It's a much harsher, more honest metric. And heavy JavaScript main-thread work is the usual culprit behind a failing score. If any tool or report you're using still references FID, it's stale; Google removed FID from Search Console the day INP took over.
Fixing each one
- LCP. Compress and correctly size your hero image, serve modern formats (WebP/AVIF), preload the LCP element, and cut slow server response times with caching and a CDN.
- INP. Break up long JavaScript tasks, defer non-critical scripts, trim third-party tags, and avoid doing heavy work in response to clicks. This is where most 2026 sites lose points.
- CLS. Set explicit width and height (or aspect-ratio) on images and embeds, reserve space for ads, and never inject content above what the user is already reading.
Layer 6. Structured data and schema
Structured data is machine-readable markup. JSON-LD is the format Google recommends. That tells search engines and AI assistants exactly what a page is: an article, a product, a local business, an FAQ, a review. It doesn't directly raise rankings, but it does two things that matter enormously in 2026: it makes you eligible for rich results (stars, FAQs, prices in the SERP), and it gives AI answer engines a clean, confident signal about what your content is and who's behind it.
- Mark up what you are. At minimum use
Article/BlogPostingon posts,Organizationsitewide, andLocalBusinessif you serve a place. AddFAQPagewhere you genuinely answer questions. - Keep it accurate and visible. Schema must reflect content the user can actually see on the page. Marking up content that isn't there is a structured-data spam violation.
- Validate everything. Run pages through Google's Rich Results Test and the Schema.org validator. Invalid JSON-LD is worse than none. It can disqualify you from rich results silently.
- Name your author and publisher. Person and Organization markup feeds the E-E-A-T signals both Google and AI engines lean on when deciding whom to trust and cite.
Layer 7. Technical AEO: making your site legible to AI crawlers
This is the layer that didn't exist five years ago and now belongs in every audit. AI assistants reach your site through their own crawlers, and they fall into three buckets that you should treat differently. Block the wrong one and you vanish from AI answers; allow the wrong one and you hand your content to a training set for free. Here's the 2026 lay of the land.
| User-agent | Operator | Purpose | Usually allow? |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Retrieval for ChatGPT search results | Yes. It can cite you |
| PerplexityBot | Perplexity | Retrieval / indexing for answers | Yes. It can cite you |
| GPTBot | OpenAI | Training data collection | Your call |
| ClaudeBot | Anthropic | Training data collection | Your call |
| Google-Extended | Gemini / Vertex AI training | Your call. See below | |
| CCBot | Common Crawl | Open dataset used by many LLMs | Your call |
Two important nuances. First, Google-Extended. Introduced on 28 September 2023. Controls only whether Google may use your content to train Gemini and other generative models. Blocking it has no effect on your normal Google Search ranking, so you can opt out of AI training while staying fully indexed in Search. Second, robots.txt is a polite request, not a wall: Cloudflare reported in August 2025 that Perplexity was using undeclared crawlers that rotated user-agents and IPs to slip past no-crawl directives, and other operators have a patchy compliance record too. If you need to truly block a bot, enforce it at the server or WAF level, not just in robots.txt.
For most businesses that want to be found and cited by AI, the move is to allow the retrieval/search bots, make a deliberate decision on the training bots, and. Critically. Make sure your content lives in clean server-rendered HTML so the bots that don't run JavaScript can read it. Pair that with an llms.txt file and solid schema and you've covered the technical side of answer-engine optimisation. The content side. Being mentioned and trusted across the web. Runs on the same authority work as classic SEO, which is exactly why the right backlinks still matter.
Layer 8. Duplication, canonicals, HTTPS and hreflang
The housekeeping layer. None of it is glamorous; all of it quietly leaks ranking signals when it's wrong.
- HTTPS everywhere. Non-negotiable and a lightweight ranking signal. Make sure HTTP redirects to HTTPS, your certificate is valid, and there's no mixed content loading insecure assets.
- One home for each page. Pick www-or-not and trailing-slash-or-not and redirect the rest with 301s. Don't let four versions of your homepage compete with each other.
- Tame parameters and facets. Sorting, filtering and tracking parameters can spawn thousands of near-duplicate URLs. Use canonicals, and where appropriate robots.txt, to keep crawlers focused on the versions that matter.
- hreflang for multi-language sites. If you serve multiple languages or regions, hreflang tells Google which version to show whom. It's fiddly and error-prone, so validate it; broken hreflang clusters are a classic silent leak.
Common technical SEO mistakes (we see these weekly)
- Blocking the whole site in robots.txt after a launch, then wondering why traffic flatlined.
- Leaving a sitewide
noindexfrom the staging build in production. - Treating a lab Lighthouse score as the Core Web Vitals verdict. The scoreboard is field data in CrUX, at the 75th percentile.
- Still optimising for FID. It was retired in March 2024; INP is the metric now.
- Letting JavaScript build critical content or error pages, then losing it to the rendering queue.
- Schema that describes content the page doesn't show. A spam signal that can kill rich results.
- Bloated XML sitemaps full of redirects, 404s and noindexed URLs, confusing the crawler about what's canonical.
- Accidentally blocking the AI retrieval bots you actually want to be cited by, while leaving the training bots wide open.
The tools you actually need
You can run 90% of a technical audit with free tools. The paid crawlers earn their keep on bigger sites.
- Google Search Console (free). Your source of truth for indexation, coverage, crawl stats, Core Web Vitals field data and URL Inspection. Start here, always.
- PageSpeed Insights / web.dev (free). Pulls both lab and CrUX field data for Core Web Vitals on any URL.
- Screaming Frog or Sitebulb. Desktop crawlers that surface broken links, redirect chains, duplicate titles, orphan pages and noindex issues at scale.
- Rich Results Test & Schema.org validator (free). Confirm your structured data parses and is eligible for rich results.
- Ahrefs or Semrush. Their Site Audit modules schedule crawls and trend technical health over time; see our honest roundup of the best AI SEO tools for which suits you.
Who needs a technical audit. And roughly what it costs
Every site benefits from a technical pass, but the urgency scales with size and stack. A small, fast WordPress brochure site might need an hour a quarter. A large e-commerce or JavaScript-app site with faceted navigation needs ongoing attention, because new technical debt appears with every release.
On cost: a one-off professional technical audit typically runs anywhere from a few hundred pounds for a small site to several thousand for a large, complex one. The range is wide because the work scales with URL count and platform complexity, so treat any single figure with suspicion and get a scoped quote. If you'd rather build the skill yourself, that's the cheaper long game: the audit workflow above is teachable, and it's the foundation we drill in The Dojo before anyone touches content or links. And if you just want the leaks found and fixed without becoming a technical SEO yourself, that's what our done-for-you work is for.
The 2026 view: where technical SEO is heading
Three currents are worth watching. First, rendering scrutiny is increasing. The December 2025 non-200 clarification is part of a pattern of Google being more explicit (and arguably less forgiving) about JavaScript-dependent content. Server-rendered, HTML-first sites are pulling ahead. Second, AI legibility is becoming a first-class concern. Managing crawler access, exposing clean facts in HTML, and using schema to make your entity unmistakable now feeds both Search and the answer engines. Third, the fundamentals are stubbornly stable: crawl, render, index, fast, clean. The acronyms change; the plumbing doesn't.
That's the reassuring part. You don't need to chase every update. Nail the eight layers above, re-check them after every significant release, and you've built a site that's ready for whatever Google and the AI engines do next.
People also ask
What is technical SEO in simple terms?
Technical SEO is the work that makes your website easy for search engines and AI assistants to crawl, render, understand and trust. It covers crawling and indexing, site speed (Core Web Vitals), mobile-friendliness, structured data and clean code. The foundation that lets your content and links actually rank.
What are the Core Web Vitals thresholds in 2026?
A "good" score means LCP under 2.5 seconds, INP under 200 milliseconds, and CLS under 0.1. To pass overall, at least 75% of real visits to a URL must fall in the "good" band, measured on field data in the Chrome User Experience Report (CrUX).
Did INP replace FID?
Yes. Interaction to Next Paint (INP) officially replaced First Input Delay (FID) as a Core Web Vital on 12 March 2024. INP measures responsiveness across the whole visit rather than just the first interaction, and Google removed FID from Search Console that day. If a report still mentions FID, it's out of date.
Why isn't my JavaScript content getting indexed?
Google renders JavaScript, but in a separate, queued step. And as of its December 2025 documentation update, pages returning a non-200 status (like a JS-built 404) may be skipped for rendering entirely. If your content, links or schema only appear after client-side JavaScript runs, server-render the important parts and check the rendered HTML in Search Console's URL Inspection tool.
Should I block AI crawlers like GPTBot in robots.txt?
It depends on the bot. Retrieval bots like OAI-SearchBot and PerplexityBot fetch pages to cite in answers, so allowing them helps you get found in AI search. Training bots like GPTBot, ClaudeBot and CCBot collect data for model training. That's a judgement call. Note that robots.txt is a request, not enforcement; if you must block a bot, do it at server or firewall level.
Does blocking Google-Extended hurt my rankings?
No. Google-Extended only controls whether Google can use your content to train Gemini and other generative models. Blocking it has no effect on how your pages are crawled, indexed or ranked in normal Google Search, so you can opt out of AI training while staying fully visible in search results.
How often should I run a technical SEO audit?
Check Search Console weekly for new errors, run a full crawl quarterly for a small site, and audit after every significant release or migration. Large or JavaScript-heavy sites need closer to monthly attention, because technical debt accumulates with each deployment.
About the author
David Sessford
Founder of The KillerEdge, with 14+ years in SEO. David builds and audits the technical foundations behind ranking sites for businesses across the UK and beyond More about David →
Want your site audited and fixed?
Join The Dojo and we'll walk you through this exact technical audit on your own site. Crawl, render, Core Web Vitals, schema and AI crawlers. And show you the fastest fixes first.
Join The Dojo. $67/mo →