Technical SEO After AI: What Changed in Search Engines and What You Should Do

What is technical SEO after AI, and what actually changed?

Technical SEO after AI is the practice of keeping a website crawlable, renderable, fast and well structured so that both classic search engines and AI answer systems can read it. The method has not changed at its core. What changed is the audience: new bots, new report types, and far fewer clicks per ranking.

I have been doing technical SEO since 2012, first for my own projects and later for clients through my SEO consulting service. However, the last two years felt different. Clients started asking about llms.txt, "AI schema" and blocking ChatGPT. Most of those questions had simple answers, and most of the answers pointed back to old fundamentals.

In short, the AI era did not invent a new discipline. Instead, it raised the price of ignoring the old one. A page that Googlebot cannot render, or a page that loads in six seconds, now loses in two places at once: the classic results and the AI answer box. This guide walks you through what I check, in the order I check it.

Do you need extra technical requirements to appear in AI Overviews and AI Mode?

No. Google's official guide for optimizing for generative AI features says a page only needs to be indexed and eligible to show a snippet in order to appear as a link in AI Overviews and AI Mode. The document states clearly that there are "no additional technical requirements".

That single sentence removes a lot of anxiety. For example, if your page already ranks and shows a description in classic results, it is technically eligible for the AI features. Therefore, the question is not "how do I qualify?" but "why is my eligible page not chosen?". That second question is almost always about content quality, clarity and structure.

In practice, I still run a technical audit before touching content. The reason is simple: a page blocked by a stray noindex, a broken canonical or a render error is not eligible at all, and no amount of rewriting fixes that. Technical SEO is the entry ticket; content decides the seat.

Are llms.txt, custom schema and "rewriting for AI" really necessary?

According to the same Google guide, they are not. Google writes that llms.txt, content chunking, AI specific rewrites and special schema.org markup are not required. On llms.txt specifically, the wording is blunt: the file neither helps nor hurts.

I know this disappoints people who sell AI readiness packages. However, I would rather you spend the budget where it moves numbers. For instance, the hours some agencies bill for an llms.txt file would cover a full render test and a Core Web Vitals fix on most small sites.

• llms.txt: harmless, but no measurable effect on Google. Skip it unless a specific platform documents support.
• Chunking content into "AI sized" blocks: unnecessary. Clear headings and short paragraphs already do this job for humans and machines alike.
• Custom "AI schema": does not exist as a Google requirement. Standard schema types remain useful for rich results, which I cover later.
• Rewriting pages "for AI": pointless as a separate task. Write clearly for people and the machines follow.

In other words, the technical SEO checklist did not grow a new chapter. It grew a new reason to finish the old chapters properly.

How do AI bots crawl your website?

AI systems reach your site through several distinct crawlers, and each one has a different job. Google uses standard Googlebot for Search, including AI Overviews and AI Mode. There is no separate AI Overviews crawler, so normal Googlebot access is enough.

OpenAI, on the other hand, documents three bots on its official crawler page. GPTBot collects data for model training. OAI-SearchBot fetches pages so that ChatGPT can show sites in its search answers. ChatGPT-User visits when a person triggers an action inside ChatGPT, and OpenAI notes that robots.txt rules may not apply to that user driven traffic.

Google also runs a product token called Google-Extended. It is not a crawler in the usual sense. Instead, it is a robots.txt token that controls whether your content may be used for training future Gemini models and for grounding in Gemini and Vertex AI. I explain the ranking implications of that token in a moment, because it causes more confusion than any other item in technical SEO right now.

Which AI bots should you allow or block in robots.txt?

My default recommendation is to allow the bots that send visibility and to decide case by case on the bots that only take. The table below summarizes the five names that matter most, based on the official Google and OpenAI documentation.

OpenAI states that sites blocking OAI-SearchBot are not shown in ChatGPT search answers, and it confirms that you can block GPTBot while allowing OAI-SearchBot. Therefore, a publisher who worries about training data can still keep the search door open. You can build the rules with my free robots.txt generator and test them before uploading.

Does blocking Google-Extended lower your rankings?

No. Google's common crawlers documentation says, in its own words, that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. Blocking it is purely a decision about Gemini training and grounding.

I get this question from nearly every publisher client, so let me be precise. If you disallow Google-Extended, Googlebot still crawls, still indexes and still shows your pages, including inside AI Overviews. Consequently, the decision belongs to your legal and business team, not to your SEO checklist.

That said, I do ask clients to think about the trade off honestly. Grounding is the mechanism that lets Gemini cite live pages. Blocking the token may reduce those citations in Gemini products outside Search, while doing nothing for your rankings. For most service businesses, I recommend leaving it open. For media companies with licensing strategies, blocking can make sense.

Which 7 checks come first in a technical SEO audit after AI?

My checklist after AI has seven steps, and I run them in this exact order because each step depends on the one before it. This is the list I use on my own sites and on every client site before any content work begins.

1. Decide bot by bot in robots.txt: Googlebot and OAI-SearchBot open, GPTBot and Google-Extended by policy.
2. Clean up noindex and canonical tags, so that every page you want in AI answers is actually eligible.
3. Run a render test: Search Console URL inspection plus a view of the page with JavaScript disabled.
4. Measure Core Web Vitals in the field: LCP, INP and CLS on your top templates.
5. Refresh the XML sitemap, remove dead URLs and make sure new pages appear within a day. My XML sitemap generator helps on small sites.
6. Validate schema for the rich result types you actually target.
7. Monitor server logs for bot behaviour, errors and crawl frequency.

Notice that none of these steps is new. However, step one and step seven changed in weight. Bot decisions used to be a footnote; now they decide which AI answer engines can see you at all. Log monitoring used to be a nice to have; now it is how you notice that an AI crawler is hammering your server without sending anyone back.

Is JavaScript generated content visible in AI searches?

Only partly, and that is one of the biggest technical SEO risks I see today. Google's JavaScript SEO documentation explains that Googlebot processes pages in three phases: crawling, rendering and indexing. Every page that returns a 200 status enters a render queue and runs in an evergreen Chromium.

So far, so good for Google. However, the same document states that other search bots may not execute JavaScript at all. In my experience that sentence describes most AI crawlers. If your product descriptions, prices or article bodies appear only after a client side script runs, an AI bot that skips rendering sees an empty shell.

The test is simple. Open your page, disable JavaScript in the browser and look at what remains. If the main content disappears, you have a visibility problem in every engine that does not render. Then confirm the Google side with URL inspection, which shows the rendered HTML Googlebot produced. Both views should contain your key content.

Server side rendering or client side rendering: which is safer now?

Server side rendering, or a static build, is the safer choice for technical SEO after AI, because the content exists in the HTML before any script runs. Google itself notes that server side rendering benefits both users and crawlers. The table compares the common approaches.

When I build sites through my web design service, I default to static or server rendered pages for anything a customer might search for. Interactive parts, for example a price calculator, can hydrate on the client afterwards. As a result, the searchable content never depends on a script finishing.

Do Core Web Vitals still matter for technical SEO in the AI era?

Yes. Google's Core Web Vitals documentation sets the thresholds at LCP within the first 2.5 seconds, INP under 200 milliseconds and CLS under 0.1, and it says these metrics align with what its core ranking systems seek to reward. Nothing about AI features changed that page.

There is also a practical reason that has nothing to do with rankings. A slow page wastes crawl budget for every bot, including the AI ones that visit far more often than they refer. Faster pages get crawled more completely and more often, which matters when you publish updates you want AI answers to reflect quickly.

In my audits, the three fixes that move Core Web Vitals most are always the same: a properly sized and preloaded hero image for LCP, less third party JavaScript for INP, and reserved space for images and ads for CLS. Measure in the field, not only in the lab, because real visitors on mid range phones are the ones Google records.

How do snippet controls work in AI searches?

The same controls you already know still work. Google's AI features and your website page confirms there is no new tool for limiting how content appears in AI features. The existing nosnippet, data-nosnippet, max-snippet and noindex controls apply as before, and you do not need any new machine readable file or markup.

This matters for technical SEO in two directions. First, if you want to appear in AI Overviews, do not accidentally restrict snippets. For example, a leftover max-snippet:0 from an old privacy project quietly removes you from the AI answer box while classic listings survive. Second, if you want to hide a section, data-nosnippet gives you paragraph level control.

• nosnippet: no text snippet at all, and therefore no AI Overview presence for that page.
• max-snippet:[number]: caps the characters Google may quote.
• data-nosnippet: an HTML attribute that hides a specific span, div or section from snippets.
• noindex: removes the page from Search entirely, including AI features.

Check your templates for these directives before you blame the algorithm. My meta tag generator produces clean robots meta tags if you need to rebuild them.

How should you read the new Search Console generative AI report?

Search Console now includes a "Search Generative AI performance" report. Google announced it in June 2026 and completed the worldwide rollout to all sites on 31 August 2026. The report separates generative AI impressions in AI Overviews, AI Mode and Discover by page, country, device and date.

However, read the limits carefully. At launch the report contains no clicks, no click through rate and no query data. In other words, it tells you where you appeared, not what people typed or whether they visited. I treat it as a presence report, not a traffic report.

Here is how I use it in practice. First, I compare AI impressions with classic impressions per page to see which templates the AI features favour. Then I check whether pages with zero AI impressions share a technical trait, for instance heavy client side rendering or a restrictive snippet directive. That comparison turns an abstract report into a concrete technical SEO to do list.

How do you justify technical SEO investment while click through rates fall?

You justify it with arithmetic, because the falling click rate is precisely why every eligible page must be perfect. Ahrefs updated its study on 4 February 2026 with 300,000 keywords. For queries with an AI Overview, the position one click through rate fell from 0.073 in December 2023 to 0.016 in December 2025, a 58 percent drop. Even informational queries without an AI Overview fell from 0.076 to 0.039. You can read the full Ahrefs update yourself.

Example calculation, using those Ahrefs figures: a page ranking first with 10,000 monthly impressions used to earn about 730 clicks at 0.073. At 0.016 the same page earns about 160 clicks. To keep 730 clicks, you would need roughly 4.6 times the impressions, so around 45,600.

That is my interpretation, not an Ahrefs claim, but the conclusion is hard to escape. You cannot quadruple impressions with one page. Therefore you need many more pages ranking, each fully crawlable, fast and eligible for AI answers. Technical SEO is how you keep that larger surface area healthy at scale.

When is structured data still valuable for AI?

Structured data remains valuable when it supports a rich result you actually want, and when it helps machines confirm facts they already see in your visible text. Google does not require any special schema for AI features. However, the ordinary types keep doing ordinary work.

For example, I still mark up articles, FAQs, products, local business details and breadcrumbs. Product schema feeds price and availability into Shopping surfaces. Organization schema helps disambiguate a brand name that AI systems might confuse with another company. Breadcrumb schema clarifies site hierarchy for any crawler.

What I stopped doing is adding schema for its own sake. Marking up content that does not appear on the page, or stuffing invented types into JSON-LD, brings no AI benefit and risks a manual action. Keep the markup honest, keep it in sync with visible text and validate it after every template change. My schema generator produces the common types without hand editing JSON.

Why are site architecture and internal links critical for AI bots?

Site architecture decides which pages any bot discovers, how often, and how much authority flows to them. AI crawlers have limited patience and, as I noted earlier, often no JavaScript. Consequently, a page that is reachable only through a script driven menu or an infinite scroll feed may never be fetched at all.

My rule is three clicks from the home page to any page that should appear in search or AI answers. Additionally, I want every important page linked from plain HTML anchors, not from onclick handlers. A crawler that reads raw HTML must see real href attributes.

• Use descriptive anchor text so that machines understand the target page before fetching it.
• Keep category hubs shallow and link to the freshest content from them.
• Avoid orphan pages; every URL in the sitemap needs at least one internal link.
• Fix redirect chains, because each hop costs a bot a request it may not spend.

Good architecture also protects against the crawl to refer imbalance I discuss next. If bots find what they need quickly, they consume fewer server resources for the same coverage.

How do you monitor AI bots in your server logs?

You monitor them by filtering your access logs by user agent and watching four numbers over time. Cloudflare's Radar analysis from 1 July 2025 gives the motivation: in one week of June 2025, Anthropic's Claude platform made roughly 71,000 HTML requests per HTML referral. Cloudflare notes the ratio may be exaggerated because in app referrals often carry no Referer header, so treat it as a signal, not a precise fact. The direction is clear anyway: AI bots crawl heavily and refer lightly, so plan server capacity accordingly.

1. Filter by user agent: Googlebot, Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User and any others that appear.
2. Track the 4xx and 5xx ratio per bot. A spike in 5xx means your server is buckling under crawl load.
3. Record crawl frequency per bot per day, and note which sections they prefer.
4. Compare with referrals and clicks from the same platforms, so you know what each bot gives back.

Once a month I export these numbers into a simple sheet. Then I decide whether a bot deserves its access, needs a crawl delay, or should be blocked. That is technical SEO as resource management, and it is a routine I recommend to every client with a server bill.

How should multilingual sites set up technical SEO after AI?

Multilingual sites need the same fundamentals applied per language, plus correct hreflang. I run talhaaslan.com in Turkish, English and German, so every check in this guide runs three times on my own domain. AI systems answer in the user's language, and they pull from pages in that language, so every version must be independently crawlable and fast.

Hreflang tells Google which version to show to which searcher. The tags need a complete set, a self reference and matching return links, otherwise Google ignores the group. My hreflang generator builds the full block for a page group, and I recommend validating it after every new language launch.

Speed also varies by language more than people expect. For instance, a German page served from an Istanbul server without a CDN can miss the LCP threshold that the Turkish page passes comfortably. Therefore, measure Core Web Vitals per country in the field data, not only on your home market. The Search Console generative AI report splits impressions by country, which makes this comparison easier.

Which free tools can you use for technical SEO?

You can complete most of this checklist with free tools, and I built several of them myself so that clients can verify my work. The paid crawlers are excellent, but a small business does not need them to get the fundamentals right.

• Search Console: URL inspection for render checks, the Core Web Vitals report, and the new generative AI performance report.
• PageSpeed Insights: lab and field data for LCP, INP and CLS on any public URL.
• The redirect checker on my site: reveals redirect chains and wrong status codes that waste bot requests.
• The Google SERP preview: shows how your title and description will look before you publish.
• Your browser: disable JavaScript, view source, and read the raw HTML like a bot would.
• Your server logs: the most honest data source you own, and usually the most neglected.

The complete tools page lists everything, all without registration. In addition, the Rich Results Test from Google validates the schema you generate. That combination covers all seven steps of the checklist with no licence fees.

What are the 6 most common technical SEO mistakes after AI?

The mistakes I see most often are not exotic. They are old mistakes with a higher price tag, because each one now removes a page from two result types instead of one.

1. Blocking OAI-SearchBot together with GPTBot, and then wondering why ChatGPT never mentions the brand.
2. Believing Google-Extended affects rankings, and leaving it open or closed for the wrong reason.
3. Shipping client side rendered content and assuming every bot renders like Googlebot.
4. Leaving an old max-snippet or nosnippet directive in a template, which silently excludes pages from AI Overviews.
5. Buying llms.txt files and "AI schema" instead of fixing LCP and broken canonicals.
6. Never opening the server logs, so AI crawler load and 5xx errors go unnoticed for months.

Every item on this list has appeared in a real audit I ran in the past year. However, I describe them as patterns rather than case studies, because the specific numbers belong to my clients. The fix for each one is in the earlier sections of this guide.

How many days does a small business need to apply this checklist?

A small business with a single site and a few hundred pages can complete the checklist in about two weeks, working a couple of hours a day. Below is an example schedule, not a promise; the exact time depends on your platform and how many problems the audit uncovers.

• Start on days 1 and 2 with robots.txt decisions, bot by bot, and a written policy for training bots.
• Days 3 to 5: noindex and canonical clean up across templates, plus a sitemap refresh.
• Next, days 6 and 7: render tests with JavaScript disabled and URL inspection on each template.
• Days 8 to 10: Core Web Vitals fixes, starting with the LCP image and third party scripts.
• Then days 11 and 12: schema validation and snippet directive review.
• Days 13 and 14: log filtering set up, first report, and a recurring monthly reminder.

If you would rather have me run it, my packages page lists what a technical audit includes, and you can contact me directly; there is no account manager in between. Either way, start with robots.txt today. It is the one file that decides who is even allowed to read the rest of your work.