Technical SEO is the work that lets a search engine reach your pages, read them, and store them. Writing the words on the page is a separate job. Technical SEO decides whether a search engine can open your page at all.
Most checklists you will find have the same problem. They were written years ago and copied since. They still tell you to use tools Google switched off in 2021. They still quote a file size limit more than seven times higher than the real one. They still tell you to add a file. Four separate studies found that nothing reads it.
We checked each rule below against Google's current documentation, and dated every change.
The checklist at a glance
Seven parts, in the order they matter. There is no point tuning your speed score if Google cannot reach the page.
| # | The check | The question it answers |
|---|---|---|
| 1 | Crawling | Can Google get to the page at all |
| 2 | Rendering | Can Google see the content once it arrives |
| 3 | AI crawlers | Can ChatGPT and Perplexity see it too |
| 4 | Core Web Vitals | Is the page fast enough for real visitors |
| 5 | Structured data | Does Google understand what the page is |
| 6 | Architecture | Can Google tell which page is the real one |
| 7 | Measurement | Are you reading the right report |
Part 1. Can Google reach your pages
Google must fetch a page before it can rank it. So start here every time.
Googlebot now reads only the first 2MB of your HTML. For years everyone quoted 15MB. In a post published on 31 March 2026, Google's Gary Illyes wrote that Googlebot "currently fetches up to 2MB for any individual URL". The 15MB figure is now only the default for crawlers that do not set their own limit.
Google does not reject a bigger page. It stops reading at 2MB. Everything after that point, in Google's words, is "entirely ignored". It is not fetched, not rendered, not indexed.
Long base64 image strings, a huge block of CSS in the head, or a giant menu rendered before your main content all eat that budget. Your real content can end up after the 2MB point, so Google never reads it.
Google's own fix is simple. Move heavy CSS and JavaScript into separate files, because each file gets its own 2MB budget. Then put the important things high in the HTML: the title, the meta tags, the canonical, and your main structured data.
How to check your HTML document size in one minute. Open your page in Chrome. Press F12. Click the Network tab and reload. Click the first row, which is the HTML document itself. Read the Size column. Under 500KB is comfortable. Near 2MB is a problem. Ignore the total page weight, because only that one document counts here.
Do not use noindex to save crawl budget. People do this constantly and it backfires. Google's crawl budget documentation tells you not to, because Google still has to crawl the page to see the noindex tag.
If you block a page in robots.txt, Google cannot read the noindex tag on it. So the page can stay in the index with no description. Pick one. Block it, or noindex it. Never both.
Three facts about robots.txt worth knowing. Google's robots.txt specification sets a 500 KiB size cap. The file is cached for about 24 hours. And a 5xx server error on the file itself stops Google crawling your site for 12 hours. That last one means a short server error can cost a day of crawling.
The Removals tool does not remove anything. It hides a page from results for about six months, then the page comes back. Google's own help page says so. It does not stop crawling and it does not delete the page. Use noindex if you want a page gone properly.
Google now publishes a page called Myths about crawling. Read it. It contradicts several things agencies still say in pitches.
Google moved all its crawling documentation to a separate site during late 2025 and early 2026. So roughly half the Google links in an older technical SEO checklist now redirect or break. If your consultant sends you a dead Google link, the checklist is old.
If crawling is your problem, our technical SEO work begins with fixing it.
Part 2. Can Google read them once it arrives
Fetching a page is not the same as seeing the content. If your site builds itself in the browser with JavaScript, rendering decides whether you rank.
The "two waves of indexing" story is out of date. You will still read that Google crawls first, then comes back days later to run your JavaScript. Google's JavaScript SEO basics no longer describes it that way. Google renders pages that return a 200 status, and for most pages it happens within seconds of crawling. Some pages take longer, but the delay is hours, not days.
Three changes in December 2025 break common practice.
- A non-200 status can stop rendering completely. If your page returns an error code, Google may not render it at all.
- A noindex tag in your original HTML can stop the render. So the old trick of sending noindex then removing it with JavaScript does not work. Google may never run the script that removes it.
- Never set or change your canonical URL with JavaScript. Google says canonicalisation runs both before and after rendering, so a JavaScript canonical creates conflicting signals.
Google no longer recommends dynamic rendering. Serving a prerendered copy to bots and the real app to humans used to be Google's advice. Google now calls it a workaround and tells you not to build new sites this way.
The replacement is server-side rendering or static generation. Send real HTML. That decision fixes this whole section, and as Part 3 shows, it matters far more in 2026 than it ever did for Google alone.
How to check what Google sees. Open Search Console, use URL Inspection, and click Test Live URL. Then open the rendered HTML and the screenshot. If your main text is missing there, Google is not seeing it either.
One warning about that test. It crawls as Google-InspectionTool, which is a different user agent from Googlebot. Blocking one does not block the other, so a clean test does not prove Googlebot can reach the page.
Part 3. Can AI crawlers read them
This part did not exist in checklists three years ago. It is now the section with the most expensive mistakes in it.
AI crawlers do not run JavaScript. Not one of the major ones. A study by Vercel and MERJ measured a month of real traffic: 4.5 billion Googlebot fetches, 569 million from GPTBot, and 370 million from Anthropic's crawler. Google rendered. The AI crawlers did not.
Googlebot runs your JavaScript, so the page can rank in Google. GPTBot does not run it, so the same page is blank to ChatGPT. If your site is a JavaScript app, you are invisible to AI answers while looking fine in Google.
AI bots do download CSS, JavaScript and images. You will read that they do not. The same study shows they do. ChatGPT's crawler spent 11.5% of its requests on JavaScript files and Claude's spent 23.8%. They simply never execute what they downloaded. The bandwidth is spent. The content never appears.
A third of their fetches are wasted on missing pages. The same study found ChatGPT's crawler spent 34.8% of its fetches on 404 pages, and Claude's 34.2%. So a stale sitemap or a batch of dead internal links costs you a third of the attention these crawlers give you. Fixing your 404s is now an AI visibility task, not just routine maintenance.
Blocking GPTBot does not remove you from ChatGPT. This is the most common error we see. OpenAI runs four separate bots and each does a different job. GPTBot is the training crawler. OAI-SearchBot controls whether you appear in ChatGPT's search results. Block GPTBot and you keep appearing in ChatGPT while giving up nothing you wanted to keep.
Blocking Google-Extended does nothing to AI Overviews. Google's crawler documentation is clear that Google-Extended is not a crawler at all. It never controlled AI Overviews. Plenty of sites added that rule and believed they had opted out.
There is a real control now. In Search Console, open Settings, then Search generative AI. Google began testing it on 3 June 2026 and rolled it out worldwide on 31 August 2026. Every property defaults to Include, so you have to switch it yourself. Google says the setting is not used as a ranking signal. And it works at domain level only, with page level control planned for March 2027. It covers AI Overviews, AI Mode and AI Overviews in Discover. It does not cover the Gemini app.
AI crawler traffic is now enormous. Cloudflare's published network figures put GPTBot at 11.7% of AI bot traffic in July 2025, and ClaudeBot at 9.9%. A year earlier they were 4.7% and 6%. Yet only a small minority of the top 10,000 sites have any AI bot rules at all.
The llms.txt file does nothing, and four studies say so
You have probably been told to add an llms.txt file so AI systems can read your site. It does not work.
| Who checked | What they found |
|---|---|
| Ahrefs, 38,360 files across 137,210 domains | 97% of those files got zero traffic in May 2026 |
| Ahrefs, same study | Of the few fetches that happened, only 19.5% came from named AI tools |
| SE Ranking, about 300,000 domains | No link between having the file and being cited more by AI |
| ALLMO.ai, 94,614 cited URLs from 11,867 AI answers | Only 1 of the 50 most-cited domains had one |
| Otterly.ai, one site, 90 days | 84 of 62,100 AI bot visits touched it, about 0.1% |
Google answered it directly. Its AI optimization guide states that Search does not use llms.txt. Google also names Markdown as a format you do not need to create, because ordinary HTML is what its systems read.
OpenAI's own crawler documentation points site owners to robots.txt and never asks for an llms.txt.
There is one honest complication. Chrome's Lighthouse does include an llms.txt check. That check belongs to its agentic browsing category, which is about software operating your page, not about Google Search. It is not a ranking signal.
What to do instead. Send real HTML from the server. It is the same fix as Part 2, and it is the one thing that helps Google, ChatGPT, Perplexity and Claude together. If you want the longer version, we wrote about how AI search differs from ordinary SEO.
Part 4. The speed that actually counts
Most speed advice measures the wrong thing.
FID is gone. INP replaced it. The swap happened on 12 March 2024, and FID was removed from every Google tool on 9 September 2024. If a report still shows you a First Input Delay number, it was generated by something that has not been updated in two years.
The three current metrics and their pass marks:
| Metric | What it measures | Good |
|---|---|---|
| LCP | How fast the main content appears | Under 2.5 seconds |
| INP | How fast the page responds when tapped | Under 200 milliseconds |
| CLS | How much the layout jumps about | Under 0.1 |
Failing one metric fails the whole page. Two out of three is not a pass.
Lighthouse does not measure INP at all. This is the trap that costs people the most money. Google's own tooling guide confirms Lighthouse measures Total Blocking Time instead, and only during page load. So a 100 out of 100 performance score tells you nothing about your INP. Agencies sell that score anyway.
The number Google actually uses comes from real Chrome visitors, not from a lab test. Open PageSpeed Insights and read the field data at the top, not the lab score below it.
Too much JavaScript is the usual cause of poor INP. Poor INP comes from long tasks on the main thread, which nearly always means too much JavaScript. If your site uses a JavaScript framework, hydration runs on the main thread and raises INP.
The HTTP Archive Web Almanac measured 16,213,084 websites in July 2025. 77% of mobile sites now have good INP. Only 62% have good LCP. Responsiveness is mostly solved. Loading the main content quickly is not.
Two tool changes to know. The CrUX Dashboard in Looker Studio was switched off at the end of November 2025 and CrUX Vis replaced it. LCP and INP also became measurable in Safari and Firefox on 12 December 2025. You can now see iPhone traffic in your own monitoring.
Speed work usually improves conversion rate rather than rankings, which is worth remembering when someone quotes you for it.
Part 5. Structured data
Schema markup tells Google what a page is about in code, so Google does not have to guess. Useful, but far less powerful than most checklists claim.
FAQ rich results are gone. As of 7 May 2026 they no longer appear in Google Search for anyone. In 2023 Google limited them to well-known government and health sites. That exception has now gone too. If you added FAQ schema for star ratings or dropdown answers, those no longer show.
HowTo rich results died on 14 September 2023. Google deleted the documentation rather than marking it deprecated.
The Structured Data Testing Tool has not existed since 2021. If a checklist tells you to use it, that checklist is at least five years old. The job split in two. Use the Rich Results Test to see whether Google can show a rich result. Use the Schema Markup Validator to check the markup is valid.
Old schema left on your site does not hurt you. Google has said this in writing more than once. You do not need a cleanup project for markup that no longer produces a rich result.
You do not need special schema for AI Overviews. Google says so plainly and warns against over-focusing on it.
The Web Almanac's 2025 SEO chapter covered 16.2 million sites. About half of those sites use structured data, and JSON-LD has clearly won as the format. Only around 2% of sites inject schema with JavaScript. That is good, because Google documents the method as a risk.
Part 6. Architecture, canonicals and two languages
The three-click rule is not a rule. It has no research behind it. It came from an opinion in a 2001 design book. Nielsen Norman Group's write-up explains that the study people cite as proof found the opposite. Users do not give up after three clicks. They give up when they stop feeling they are getting closer.
So stop counting clicks. Make sure every page you care about is linked from somewhere a crawler will reach, and that the link text describes the destination.
Google treats a canonical tag as a hint. Google says this in plain words and regularly picks a different canonical from the one you declared. The Page indexing report in Search Console shows you when Google has picked a different one.
Google publishes a list of common canonical mistakes and most real errors are on it. Google also added guidance during late 2025 and 2026 specifically about canonical tags and JavaScript, because frameworks were producing duplicate and conflicting tags.
If you run English and Arabic, this is the bug to look for. Your canonical tags and your hreflang tags have to agree. The classic failure is an Arabic page that canonicalises to the English page. That tells Google the Arabic version is a duplicate and should not rank at all.
There is no free report for hreflang errors any more. Google removed the International Targeting report in 2022 and never replaced it.
rel=next and rel=prev have been dead since 2019. Google's current pagination advice is the opposite of what old guides say. Use ordinary crawlable links between pages, and give each page its own self-referencing canonical.
nofollow stopped being a crawl control on 1 March 2020. Google changed it to a hint. Using it to steer crawling is unreliable by design.
For filters and faceted URLs, Google reordered its advice in December 2025. Robots.txt disallow now comes first on Google's list. Google calls canonical tags and nofollow less effective in the long term.
If you are planning a rebuild, the migration side of this is covered in our guide to redesigning without losing traffic.
What stopped mattering
Every row below is something people still do.
| What you were told | Status | Since |
|---|---|---|
| Add an llms.txt file for AI | Does nothing for Google Search | Confirmed June 2026 |
| Add FAQ schema for rich results | Rich results removed for everyone | 7 May 2026 |
| Add HowTo schema | Rich results removed | 14 Sept 2023 |
| Use the Structured Data Testing Tool | Tool does not exist | 2021 |
| Run the Mobile-Friendly Test | Test retired | 1 Dec 2023 |
| Quote the 15MB crawl limit | Googlebot reads 2MB | Disclosed 31 Mar 2026 |
| Chase a First Input Delay score | Metric deleted | 9 Sept 2024 |
| Use rel=next and rel=prev | Unsupported | 2019 |
| Use nofollow to control crawling | Downgraded to a hint | 1 Mar 2020 |
| Set URL parameters in Search Console | Tool removed | April 2022 |
| Use dynamic rendering for JS sites | No longer recommended | Current docs |
| Block Google-Extended to exit AI Overviews | Never worked, not a crawler | Always |
| Follow the three-click rule | Never had evidence | Opinion from 2001 |
| Sell a 100/100 Lighthouse score | Excludes INP entirely | Current docs |
Part 7. The free tools worth using
You do not need paid software for any of this.
| Tool | The question it answers | The common misreading |
|---|---|---|
| Search Console, Page indexing | Which pages Google stored, and why not | Indexed does not mean visible |
| URL Inspection, Test Live URL | What Google sees on the page right now | The default view is the stored copy, not live |
| PageSpeed Insights, field data | Are real visitors having a fast experience | People read the lab score instead |
| Chrome DevTools, Network tab | How big is my HTML document | People read total page weight |
| Rich Results Test | Can Google show a rich result | Eligible is not the same as it will appear |
| Schema Markup Validator | Is my markup valid | It does not tell you if Google supports the type |
| CrUX Vis | How my speed changed over time | Replaced the CrUX Dashboard in Nov 2025 |
Four things about Search Console specifically.
There is a new Generative AI report. Google launched it in June 2026 and it shows where your pages appear inside AI Overviews and AI Mode. There is no click metric in it, because it reports impressions only.
You can now see the last 24 hours. Search Console added recent data so you no longer wait two or three days to see whether a fix worked.
The three main reports will never agree with each other. Search Console reports index status by canonical URL. Crawl Stats and Core Web Vitals report by the URL actually requested. Google documents this. People waste weeks trying to reconcile them.
The Links report is a sample. It is not a full list, it groups by root domain, and it will not tell you whether a link is nofollow.
How to run this in one afternoon
Do these steps in order. Stop at the first broken thing and fix it before moving on.
- Open Search Console and read the Page indexing report. Anything under Not indexed that should be indexed is your first job.
- Run URL Inspection with Test Live URL on your most important page. Check your main text is in the rendered HTML.
- Open that page in Chrome DevTools, Network tab, and check the HTML document is well under 2MB.
- Open PageSpeed Insights and read the field data at the top. Ignore the score.
- Check your canonical tags on one page per template. If you run two languages, check the Arabic page does not canonicalise to English.
- Read your robots.txt out loud. Look for anything blocked that you also want out of the index, because that combination does not work.
- Delete nothing from your old schema. It is not hurting you.
If you would rather someone else did it, our technical SEO service works through the same list. If your site is being rebuilt anyway, it is cheaper to fix this during a web design project than after it.




