Technical SEO services for Googlebot and AI crawlers

What technical SEO is, and how it differs from on-page SEO

On-page SEO is about what the page says. Technical SEO is about whether any of it reaches a machine intact. It answers four questions in order, and the order matters: a failure at one step makes every later step irrelevant.

• Fetch. Does this URL return a usable response to every agent that matters, not just to your browser?
• Read. Is the content in the HTML that came back, or does it exist only after JavaScript runs?
• Resolve. When the same content sits at several addresses, which one accumulates the signals?
• Retain. Does the page stay indexed, stay fast for real users, and stay reachable after the next release?

Content and links compound on top of that stack; they do not substitute for it. A URL that returns 403 to a crawler is worth nothing.

The engagement is for sites where one of those steps is already failing or about to: a JavaScript-built site whose server response is thin, a site going through a migration, an estate large enough that Google’s crawl-budget guidance applies, or a business that ranks on Google and is missing from AI answers. If your pages fetch, index and stay fast, the constraint is elsewhere, and we will say so.

Technical SEO is now two jobs, because there are two kinds of crawler

Google documents a rendering stage. Google’s JavaScript SEO basics says Google “processes JavaScript web apps in three main phases” — crawling, rendering and indexing — and that “Googlebot queues pages for both crawling and rendering”, so JavaScript-injected content is eventually seen, with a delay. A decade of front-end frameworks assumes that pipeline exists.

The retrieval crawlers behind AI answers are documented differently. We read all four operators’ crawler pages on 7 September 2026 — OpenAI’s, Anthropic’s, Perplexity’s and Common Crawl’s. Each names its agents and the robots.txt rules that control them, and three of the four publish user-agent strings and IP ranges.

None of the four describes a rendering step. That is an absence in the documentation, not a proven absence in the software, so we treat it as an engineering assumption and test it per site: assume the first response is what gets read.

Googlebot
AI retrieval crawlers
GPTBot · OAI-SearchBot · ClaudeBot
PerplexityBot · CCBot

1. Fetch the HTML
2. Queue for rendering
3. Render in a
headless browser
4. Index what was
rendered
1. Fetch the HTML
2. Read what came back
No rendering stage in
the documentation
3. Use what was read,
as it arrived

Search agents and training agents are separate things, and this page keeps them apart. The split is why “we rank on Google but no AI engine cites us” should be tested as a technical question before it is treated as a content one. Which discipline you need once the page is readable is the job of AI Search SEO.

What we found fetching 2,596 Indian websites the way a retrieval crawler does

Our AI Search Readiness Benchmark probed every Indian business with an official website in Wikidata — 4,116 rows collapsed to 2,596 hosts — on 3 September 2026: one request each to the home page and robots.txt, 1.5 seconds between hosts, redirects followed to a final host, server-delivered HTML only. Separately, 46 hosts answered a permanent redirect the client recorded rather than followed; those appear in the table below.

Reading nothing but what the server returns is normally a weakness. Here it is the argument, because it simulates a crawler with no documented rendering stage. Two limits travel with every figure, and the report states both: the Wikidata frame over-represents companies notable enough to have an entry, and every “absent” count is an upper bound.

648 of the 2,596 hosts — 25% — never returned a usable response. One category is not a failure at all: 21 hosts disallowed every agent in robots.txt, so they were not fetched, by design, and are counted as unreachable.

The reachable 1,948 were not much better:

• 701 — 36.0% — carried a self-referencing canonical. 438 declared an Organization entity without one, so the entity exists but the engine must guess which URL it belongs to.
• 272 — 14.0% — had no robots.txt at all; 1,288 (66.1%) declared a sitemap in it.
• 1,941 were served over HTTPS, and 23 carried a home-page noindex, which can be a misconfiguration rather than a decision.

An independent crawl measures some of the same ground, and the overlap is narrower than it looks. The 2025 Web Almanac SEO chapter, under its heading “AI crawlers named in robots.txt”, reports that “gptbot appears in 4.5% of desktop sites and 4.2% of mobile sites”. Appearing is not being blocked — a file that names GPTBot in order to allow it counts inside that 4.2% — so the two are not the same measurement. Ours is the narrower one, a root-level disallow of GPTBot, OpenAI’s training agent: 102 of 1,948 reachable hosts, 5.2%.

Which agent is blocked matters more than how many are. OpenAI’s documentation separates GPTBot, which collects content that may be used to train its models, from OAI-SearchBot, which governs whether a site appears in ChatGPT’s search results. Blocking the first is a training opt-out. Blocking the second means your pages are not shown in ChatGPT’s search answers, though they can still appear as navigational links — and only 4 hosts in the census, 0.2%, do that. A single “AI crawler blocking” rate hides the difference.

The Almanac’s other useful number is the rendering gap itself: rel=canonical in 64.27% of raw mobile HTML against 66.11% of rendered mobile HTML, roughly 1.8 percentage points. It is not comparable to our 36.0% and we will not pretend otherwise — the Almanac counted any rel=canonical, we counted an exact-match self-referencing one.

Crawlability is not indexation

Crawlable means the request succeeds and robots.txt permits it. Indexed means the engine has decided to store the page. Conflating the two produces the wrong fix, three ways.

• robots.txt is the wrong tool for removing a page from the index. The engine must fetch the page to see a noindex directive, so blocking the fetch guarantees the directive is never read.
• A missing page is a log question before it is a dashboard question. Establish whether the URL was ever fetched successfully.
• Zero impressions does not mean not indexed. It means nobody searched anything the page was shown for. We check index status directly rather than inferring it from a traffic report.

Self-referencing canonicals and duplicate resolution

Most sites serve the same content at more than one address without meaning to: with and without a trailing slash, at www and apex, on HTTP and HTTPS, with tracking parameters, through a print view. Left alone, the engine picks a canonical for you, and it may not pick the URL you promote and measure. Because a canonical is a hint rather than a directive, it has to agree with everything else. We check that five things point at one address:

• the rel=canonical tag on the page;
• the internal links that point to it;
• the XML sitemap entry;
• the url property on the Organization entity;
• the final destination of any redirect chain that reaches it.

Contradiction among those five is a common canonicalisation defect, and dashboards show only its symptom: Search Console’s Page indexing report lists the page as “Duplicate, Google chose different canonical than user”.

Core Web Vitals: the thresholds, the percentile, and the data source

The thresholds, from web.dev:

Three details do more work than the numbers themselves:

• the threshold is measured at the 75th percentile of page loads, so a fast median with a slow tail still fails;
• it is assessed separately for mobile and desktop;
• it comes from field data — the Chrome UX Report’s “anonymized, real user measurement data” — not from a Lighthouse lab score.

Now a correction, including to what this page itself used to claim: red Core Web Vitals are not an automatic demotion. Google’s page experience documentation says “Core Web Vitals are used by our ranking systems”, but also that “There is no single signal”, and that “Google Search always seeks to show the most relevant content, even if the page experience is sub-par”.

For scale, the Almanac’s June 2025 measurement had 48% of mobile and 56% of desktop pages achieving good overall Core Web Vitals. We fix vitals because slow, jumpy pages lose users — not to promise a ranking change nobody can isolate. The mobile cut is scoped under mobile SEO.

Crawl budget: whether you need it at all

Crawl budget is the first line item to question. Google’s crawl-budget guide opens by telling most readers to leave: “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” It names three kinds of site it is written for:

• 1 million or more unique pages, changing about weekly;
• 10,000 or more unique pages, changing daily;
• sites with a large portion of their URLs classified by Search Console as “Discovered – currently not indexed”.

Google calls those counts “a rough estimate”, not exact thresholds. We use them as a disqualifier: a 400-page estate with a clean Page Indexing report does not have a crawl-budget problem, and we take the line item out. Where it does apply, the work is closing infinite parameter spaces, removing soft 404s, collapsing redirect chains, keeping sitemaps accurate, and fixing the slow responses that make a crawler back off.

JavaScript SEO for React, Next.js and Vue sites

A React, Next.js or Vue build is not a problem in itself. Shipping an empty HTML shell is. The test is mechanical and you can run it yourself: request the URL with a plain HTTP client, JavaScript disabled, and read what comes back.

Rendered in a browser
Fetched with JavaScript disabled

what an agent with no rendering stage reads

<h1>Steel pipe fittings</h1>
<p>Manufactured in Rajkot since 1998…</p>
<nav> — 34 real <a> elements
<link rel=”canonical”>
<div id=”root”></div>
<script> — the entire application

No headline, no copy, no links, no canonical.

If the headline, body copy, internal links and canonical are absent, every agent without a rendering stage sees an empty page. The fixes are architectural:

• server-side rendering or static generation for pages that need to be found;
• real <a href> elements for navigation — a link that exists only as a click handler is not a link to any crawler;
• canonical and title tags emitted server-side, not patched in by a client-side head manager;
• content behind tabs and accordions present in the DOM rather than fetched on interaction.

Where full server-side rendering is out of scope, prerendering the important routes is a legitimate intermediate step, and we will say which routes are worth it.

HTTP status errors, redirect chains, and the 25% with no usable response

The failure table above reads as a checklist, because each entry has an index consequence. Google’s HTTP status codes reference states that URLs “already indexed and return a 4xx status code are removed from the index”, and that with 5xx errors “already indexed URLs are preserved in the index, but eventually dropped”. Unresolved DNS, invalid TLS, refused connections and timeouts produce no status code at all.

The 127 hosts that returned HTTP 403 are the failure most likely to be invisible to the owner. A 403 to a named research agent can be bot mitigation rather than deliberate policy — but the census sent one request from one user agent, so it records a symptom, not a cause. On an engagement we establish the cause:

• fetch the same URL as several named agents and as a browser user agent, and compare the responses;
• read the edge configuration — WAF rules, bot management, rate limits — where you can give us access;
• confirm the finding in your own logs before anyone changes a rule.

Redirect chains are the quieter version of the same problem: each hop costs latency for users and a fetch for crawlers, and chains accumulate through migrations. We collapse every redirect to a single hop, resolve loops, and convert soft 404s into honest status codes.

Log file analysis: what your server actually recorded

Every tool above infers crawler behaviour. Server logs record it, and answer four questions nothing else can:

• which agents actually requested your site, and how often;
• which URLs they spent their requests on, and what status code each received;
• whether GPTBot or PerplexityBot has ever fetched the page you want cited;
• whether your valuable templates are crawled at all, or the requests go to parameter noise.

The constraint is access: origin logs are often incomplete when a CDN sits in front, so we usually need edge logs too, and we tell you before the engagement starts whether your stack can produce them.

Site migration SEO

Migrations change every URL signal at once, so technical SEO has to start before the move rather than after it. The sequence that protects traffic is unglamorous:

• an inventory of live URLs and their current performance, taken before anything changes;
• a one-to-one redirect map — every old URL assigned a real destination, no orphan redirects to the home page;
• a staging crawl asserting status codes, canonicals, titles and structured data against that map;
• sitemap and robots.txt prepared for the new structure;
• a post-launch window watching index coverage, status codes and log activity, rather than waiting for a ranking report to reveal the damage weeks later.

Where technical SEO stops

• A one-time diagnostic with a written deliverable is a different product: deep SEO audits. This page is the ongoing remediation that follows one.
• Mobile-first indexing and the mobile cut of Core Web Vitals belong to mobile SEO.
• Hreflang and the ccTLD-versus-subdirectory decision belong to international SEO. We implement hreflang inside those engagements; we do not sell it as a line item here.
• Making one page’s answer cleanly extractable is AEO. Being corroborated and cited across the wider web is GEO.

How the work runs: remediation, not a punch list

The first two weeks are discovery on your stack: a crawl, an agent-by-agent fetch test, log access if it exists, and a recorded baseline. From there the work is ticketed by what blocks indexation first, what costs users second, and what is cosmetic never. Every fix then carries a before-and-after artefact — the HTTP response, the rendered HTML, the index status, the field measurement, or the log line showing the crawler returned and got a different answer. If we cannot produce the artefact, we do not claim the fix.

What we take out of scope

• Crawl-budget optimisation below Google’s stated scope. No large body of rapidly changing pages and no backlog in “Discovered – currently not indexed” means no crawl-budget line item.
• A ranking increase attributed to Core Web Vitals. We can move the metrics and show the field data. Attributing a rank change to that alone is not something anyone can evidence.
• A verdict on your framework without the evidence. We will not tell you React or Next.js is your problem until we have shown you the JavaScript-disabled response.
• A crawl simulation sold as log data. Where a stack cannot produce origin or edge logs, the log analysis is unavailable, and we will not substitute a crawler’s guess at crawler behaviour.

Questions buyers ask

What is technical SEO, and how is it different from on-page SEO?

Technical SEO is the work of making a website fetchable, readable and resolvable by search engines and AI crawlers: server responses, robots directives, rendering, canonicalisation, status codes, speed and index management. On-page SEO is the work of making the content match what a searcher wants. Technical SEO decides whether the page can be seen at all; on-page SEO decides whether it deserves to win once it is.

How much does a technical SEO audit cost in India?

Technical SEO audit quotes in India vary enormously and are rarely comparable, because “audit” covers everything from an automated crawl export to a manual review with a remediation plan. SEOIndia publishes a figure for the one-time audit on its deep SEO audits page, and a published starting rate for the managed monthly programme, with scope and price confirmed in a written proposal before anything is billed. When comparing quotes, ask what artefact you receive and whether implementation is included.

How long before technical fixes show up in rankings?

No honest single number exists for how long technical fixes take to reach rankings, and any agency quoting one is guessing. What can be observed on a schedule is the chain: the corrected response is live immediately; a recrawl of the affected URLs shows up in server logs and Search Console within days to a few weeks; index status changes once that recrawl is processed. Ranking change, if the technical defect was the constraint, follows the recrawl rather than the deployment.

Do I actually need crawl-budget work, or is my site too small for it to matter?

Google scopes crawl-budget management to three kinds of site: 1 million or more unique pages changing about weekly; 10,000 or more changing daily; or a site with a large portion of its URLs classified by Search Console as “Discovered – currently not indexed”. It calls those counts a rough estimate rather than exact thresholds. A few thousand stable pages with a clean Page Indexing report almost certainly have no crawl-budget problem. The exception is any site generating unbounded URLs through faceted navigation or parameters.

Are Core Web Vitals a ranking factor, and how much do they really move rankings?

Core Web Vitals are used by Google’s ranking systems as part of page experience, but Google also states there is no single page-experience signal and that Search still aims to show the most relevant content even when page experience is sub-par. Good vitals will not rescue a page that does not deserve to rank; bad vitals will not remove one that clearly does. The thresholds are LCP 2.5 seconds or less, INP 200 milliseconds or less and CLS 0.1 or less, at the 75th percentile of Chrome UX Report field data, mobile and desktop separately.

My site is built in React or Next.js — is that a problem?

A React, Next.js or Vue build is not a problem by itself. It becomes one when the server returns an empty HTML shell and every piece of meaningful content arrives only after JavaScript executes. Googlebot has a documented rendering phase and will usually catch up; the crawler documentation published by OpenAI, Anthropic, Perplexity and Common Crawl describes no rendering step, so client-only content may never reach those agents. Request the page with JavaScript disabled and check what is in the response.

Why does my site rank on Google but never get cited by ChatGPT or Perplexity?

Ranking on Google while never being cited by ChatGPT or Perplexity can mean the two pipelines are reading different pages: Googlebot renders JavaScript before indexing, so it finds content that exists only after hydration, while the AI retrieval crawlers document no rendering step. Blocking is rarely the cause — in our census of 1,948 reachable Indian business home pages, only 4 disallowed OAI-SearchBot, the agent that governs appearing in ChatGPT search. Fetch your own page with JavaScript disabled before assuming a content or authority problem.

Start with a measurement, not a proposal

We would rather show you a fetch than describe one. The free audit reads the HTML your server returns, without running JavaScript, and reports what came back, including the status code, canonical tag, robots meta and structured data. It is an automated first look, not the agent-by-agent fetch test or the log review described above; those open an engagement, and they can conclude that your technical foundation is fine.