You Shipped the Website. Now Can AI Find It?

From Vibe Coding to AI-Ready Architecture What Software Engineers Must Know in 2026

So you’ve shipped the website. The client is happy. The design looks great on mobile, Lighthouse scores are solid, and maybe you even vibe coded your way through a few components with an AI assistant helping you scaffold the whole thing in record time. Job done, right?

Not quite.

Here’s the question that most engineers don’t ask until it’s too late: Can an AI actually find what you built, and will it lead people there?

We are living through a fundamental shift in how information is discovered on the internet. The search bar as we have known it for two decades, you type a query, Google returns ten blue links, you click is on it journey of getting replaced replaced. Not gradually. Rapidly. Google officially announced recently that Search is now switching to AI mode. AI Overviews now reaches 2.5 billion monthly users, AI Mode tops 1 billion, and the era of the “ten blue links” is effectively over. ChatGPT doubled to 800 million weekly users in eight months and AI adoption jumped from 14% to 29.2% in just six months.

This changes everything about how we build for the web, not just for SEOs and marketers, but for engineers. Because the same AI assistant that helped you build the website is now the gatekeeper deciding whether your users can find it or not.

The Shift: From SEO to GEO

I wrote about this transition in an earlier writing the move from traditional Search Engine Optimisation to what is now increasingly called Generative Engine Optimisation (GEO).

Traditional SEO was largely a post-launch activity. You built the site, then someone went in and sprinkled in keywords, meta descriptions, and backlinks, and hoped Google’s crawlers would eventually reward you(Which has worked for us for decades). The ranking signals were opaque, the timelines were long, and the tactics often had little to do with how the site was actually built.

GEO is different. It is not something you bolt on after the product is done. It is a design concern from the very first line of code.

Generative Engine Optimisation is the practice of optimising your website and digital content so that AI-powered search engines including ChatGPT, Perplexity, Google AI Overviews, Gemini, and Claude cite, reference, and recommend your brand in their AI-generated answers. Unlike traditional SEO, which focuses on ranking in a list of blue links, GEO focuses on making your content the source that generative AI models pull from when answering user queries.

Where traditional SEO asked: “Is Google ranking my page?”, GEO asks: “Is the AI citing me?”

The distinction matters enormously. AI-referred visitors convert at 14.2% on average versus 2.8% for Google organic, because the buyer arrives pre-informed and recommendation-primed. The AI has done the leg work. and the heavy lifting for you. It has synthesised information, compared options, and pointed the user directly at a solution. If your site is not in that answer, you are invisible to a user who is already ready to act.

What AI Actually Does When It Encounters Your Website

To understand what you need to build, you have to first understand how these systems work when they encounter your site.

When a user asks an AI assistant, say, ChatGPT or Claude a question like “Find me a good product design agency in Birmingham”, the assistant doesn’t just reach into its training data and guess. Modern AI assistants operate in two modes: retrieval from training data (what the model learned before its knowledge cutoff) and real-time web search (what it goes and finds in real time). For anything time-sensitive or locally specific, the assistant will use its search tools to crawl the web in real time. You can read more about how ChatGPT approaches this in OpenAI’s announcement of ChatGPT Search.

This is where your website comes into the picture. The assistant sends out a crawler or uses a search API to fetch information. What it encounters on your site, how it is structured, whether it is readable, what signals of authority and clarity it carries determines whether the AI can extract useful, citable information or whether it bounces off a wall of JavaScript-rendered content it cannot parse. AI chatbots like ChatGPT, Google Gemini, Grok, and Claude are actively trying to understand your website but your beautifully designed site with its JavaScript magic and interactive elements is, to them, a maze of confusion.

And here is the critical thing: you can see this happening in your own analytics. If you look at your server logs maybe for example by user agent, you will start spotting entries like ChatGPT-User/2.0, ClaudeBot, PerplexityBot, or OAI-SearchBot. These are not random visitors. When you spot GPTBot, ClaudeBot, PerplexityBot, or any of the newer strings in your server access logs, you know an AI model is indexing, scraping, or quoting your page. If they can read your site cleanly, your content ends up in the answer. If they cannot, you do not exist yet.

The llms.txt File: Your AI-First Sitemap

The most concrete and immediately actionable thing you can do as an engineer is implement an llms.txt file(This is still emerging because of AI so not all AI crawlers supports this)

Proposed in 2024 by Jeremy Howard of Answer.AI, llms.txt is a new open-standard convention designed to help large language models navigate website content with greater precision, a plain-text file written in Markdown placed in a site's root directory that provides a concise map of a site's most important resources. Think of it as the AI-era equivalent of robots.txt and sitemap.xml combined, but designed specifically for large language models to read and understand.

The problem it solves is a real one. Your beautifully designed website with its interactive components, JavaScript-rendered content, dynamic routes, and layered navigation is, from an LLM’s perspective, a maze. llms.txt acts similarly to robots.txt and XML sitemaps to allow crawling and readability of entire websites, putting less of a resource strain on LLMs and it offers an additional benefit: full content flattening. llms.txt gives AI systems a curated, low-cost, high-signal map.

Here is a minimal example of what an llms.txt file looks like:

# Acme Software

> We build AI-powered tools for operations teams in logistics and supply chain.

## Product
- [Platform Overview](https://acmesoftware.io/platform): What the product does and how it works
- [Pricing](https://acmesoftware.io/pricing): Plans and pricing for teams and enterprises
- [API Reference](https://acmesoftware.io/docs/api): Full API documentation for integrations

## Use Cases
- [Warehouse Automation](https://acmesoftware.io/use-cases/warehouse): How we help warehouse teams
- [Route Optimisation](https://acmesoftware.io/use-cases/routing): Real-time routing for delivery fleets

## About
- [Team](https://acmesoftware.io/about): Who we are
- [Contact](https://acmesoftware.io/contact): Get in touch

That is it. Clean Markdown. A brief description of who you are, followed by links to your most important pages with short, intent-clear labels. llms.txt files can require a fraction of the tokenisation cost of a standard webpage, making them well suited to the agentic era of the web which directly increases the likelihood they use your content in answers.

As of October 2025, Google has indexed between 30,000 and 60,000 llms.txt files globally. Yoast, the most widely used SEO plugin for WordPress, is now the first major SEO plugin to natively generate and maintain llms.txt. The standard is gaining adoption quickly, and the businesses that implement it early are earning a structural advantage in AI discoverability.

The naming convention matters: the file must be named specifically llms.txt (not llm.txt) to ensure cross-platform compatibility and AI discoverability, and must be placed in the root directory. Think of it as the three-file standard for complete site discoverability: robots.txt for access control, sitemap.xml for traditional search, and llms.txt for AI tools.

AI Crawlers, User Agents, and Your robots.txt

Every major AI platform operates distinct crawlers, each with its own user agent string and a distinct purpose. Understanding the difference is important because as an engineer, you have control over which ones you allow or block in your robots.txt — and the decision you make has real downstream consequences for your AI discoverability.

AI-related bots generally fall into three categories: training data crawlers that collect web content at scale for model training, search index crawlers that power AI search results, and user-triggered agents that fetch content in real time on behalf of a human mid-conversation.

Here is a breakdown of the major ones you need to know:

OpenAI

• GPTBot — OpenAI's training data crawler. It indexes web content to improve future versions of GPT models. Blocking it prevents your content from being used in model training.
• OAI-SearchBot — The crawler used to power ChatGPT Search. Allowing this is what gets your content cited in ChatGPT search results.
• ChatGPT-User / ChatGPT-User/2.0 — The user-triggered agent that visits pages in real time when a user asks ChatGPT to look something up. This is what drives referral traffic from ChatGPT to your site.

Anthropic (Claude)

• ClaudeBot — Used for chat citation fetching and general web crawls that support Claude’s responses.
• anthropic-ai — Bulk training crawler. Distinct from ClaudeBot.
• Claude-SearchBot — The agent that powers Claude's web search capabilities.

Perplexity

• PerplexityBot — The main crawler for Perplexity's index.
• Perplexity-User — The real-time, user-triggered fetch agent.

Google

• Google-Extended — Google introduced Google-Extended on September 28, 2023 as the user-agent you use to control whether Google can use your content to train Gemini and other generative models. Disallowing this tells Google not to use your content for Gemini model training, while still allowing standard Googlebot for search ranking.

Meta / ByteDance

• FacebookBot and Bytespider — It is worth noting that Bytespider has been reported to ignore robots.txt directives, and blocking at the server or WAF level is the only real control if you want to reliably exclude it.

A practical robots.txt configuration for a business that wants AI visibility across the major platforms looks like this:

# Allow OpenAI search and user agents

User-agent: OAI-SearchBot

Allow: /
User-agent: ChatGPT-User
Allow: /

# Allow training (or disallow if you prefer)
User-agent: GPTBot
Allow: /

# Allow Anthropic
User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Allow Perplexity
User-agent: PerplexityBot
Allow: /

# Control Google Gemini training
User-agent: Google-Extended
Allow: /

# Block aggressive scrapers
User-agent: Bytespider
Disallow: /

The strategic nuance here is the distinction between training bots and assistant bots. Training bots like GPTBot and CCBot crawl websites to collect data to train AI models (There are cases that you might not want to hand over you propriety data for free its your call to make). Assistant bots like ChatGPT-User and ClaudeBot fetch content in real time when users ask questions, potentially driving referral traffic to your site. If your content ends up in the training data, it can improve how accurately future models understand and represent you and allowing the assistant bots is what generates actual referral traffic.

As of July 2025, AI bots top the list of user agents referenced across popular sites, with almost 21% of the top 1,000 websites having rules for ChatGPT’s GPTBot in their robots.txt file. You can check and verify which AI crawlers are currently visiting your site at CrawlerCheck's AI Bots directory, which catalogues over 34 active AI crawlers with detailed profiles and suggested robots.txt rules.

Structured Data and Schema Markup: Teaching AI What Your Content Means

Beyond llms.txt and robots.txt, there is a deeper layer of AI readiness that lives in how your pages are marked up. This is where the engineering decisions you make at build time directly determine how well AI systems can extract, interpret, and cite your content.

Schema markup structured data using schema.org vocabulary, typically implemented as JSON-LD, is the mechanism by which you tell AI crawlers not just what your content says, but what it means. Without schema markup, AI must guess context; with proper markup, you guide the system directly. An AI system should never have to infer from raw text whether a given page is about a product, a person, a service, a how-to guide, or an FAQ.

Content with proper schema markup shows 30–40% higher visibility in AI-generated answers. This is not marginal. It is the difference between being cited and being invisible.

The most impactful schema types for GEO are:

FAQ Schema: If your content includes questions and answers, marking them up with FAQPage schema allows AI systems to extract those Q&A pairs directly, improving visibility in AI Overviews and generative summaries.

HowTo Schema: Step-by-step content marked up with HowToStep is easily machine-parsed and frequently cited by AI systems when a user asks how to do something. This is especially valuable for documentation, tutorials, and instructional pages.

Organisation Schema: Defines your company’s identity: name, description, URL, contact information, social profiles. This is foundational. Consistent naming, location data, and structured data markup across web properties help generative models identify and distinguish an entity accurately — and when descriptions conflict across sources, the result is often a hedged or absent mention in AI-generated responses.

Product and Offer Schema: If you sell something, machine-readable pricing, features, and availability matter. Agentic search tools like OpenAI’s Operator, launched January 2026, go beyond answering questions, they browse the web, compare options, and complete tasks on behalf of users, making content with structured, machine-readable information increasingly important for inclusion in agent-driven workflows.

Article Schema: For blog posts and editorial content. Defines author, publication date, last modified date. AI platforms prefer content that is 25.7% fresher than content cited in traditional search results, so freshness signals matter enormously here.

The implementation approach is JSON-LD inside a <script type="application/ld+json"> tag in your page <head>. It does not affect the visible rendering of your page at all, it is purely machine-readable metadata that travels alongside your content. There is no performance cost and no UX trade-off. It is purely upside for discoverability.

Content Architecture: Writing for Machine Retrieval

Structured data addresses the metadata layer. But the content itself, how it is written and structured is equally important for GEO.

AI systems use Retrieval-Augmented Generation (RAG), in which a query is used to retrieve relevant document segments from an external index, and those segments are incorporated into the model’s context window before a response is generated. In these systems, visibility depends not only on the model’s pre-trained knowledge but also on whether a given document is indexed, whether its content is semantically close to the query, and whether its text is structured in a way that facilitates extraction of discrete, citable claims.

What this means practically is that vague, brand-voice-heavy, long-winded content performs poorly in generative search. Tight, direct, claim-dense content performs well. The GEO principles that matter most are:

Lead with the answer: Structure content with direct answers in the first 40–60 words of each section. Do not bury it under paragraphs of context-setting. If someone asks what your product does, the first sentence of your product page should answer that directly.

Structure with headings: Clear H1 > H2 > H3 hierarchies with one topic per section make it easy for AI systems to navigate to the right segment for a given query. Think of your headings as navigation for machines, not just visual breaks for humans.

Use statistics and specific claims: One study analysing 10,000 real-world queries found that pages with structured lists, quotes, and statistics had 30–40% higher visibility in AI responses. Maintain fact density with statistics every 150–200 words. Vague generalisations are hard to cite. Specific, attributed facts are easy to cite.

Cite authoritative sources: AI systems evaluate trust signals. The Princeton and Georgia Tech KDD 2024 study found that expert quotation addition increased AI visibility by 41%, statistics by 32%, and authoritative source citations by 30%. Content that links to authoritative external sources and quotes experts is assessed as more credible and more likely to be surfaced.

Write for consistency across platforms: When AI models find conflicting descriptions of your brand, product, or service across different pages and platforms, they hedge or omit your mention entirely. Consistent entity naming, the same company name, product names, and category descriptors everywhere helps generative models identify and accurately represent you.

APIs in an Agentic World: Design for Machine Use

The AI-readiness conversation does not stop at the website. If you are building or maintaining APIs, the same principles apply and the stakes are even higher.

We are entering an era where AI agents and automation are becoming the primary consumers of APIs. OpenAI Operator, Anthropic’s Computer Use, Google’s Project Astra, etc these are systems that complete multi-step tasks on behalf of users by calling APIs, navigating interfaces, and chaining together operations. Your API is no longer just a contract between your frontend and backend. It is an interface that AI agents will call programmatically. Verified AI agent traffic grew more than 6,900% year-on-year in 2025, and agentic interactions skew heavily commercial over 85% are product-related, with conversions from agent-recommended results materially higher than traditional search. That trajectory is not slowing down.

This fundamentally changes how you should think about API design. The traditional pattern of embedding multiple steps in a single route because it is convenient, because it reduces network round-trips, because it feels “clean” from a developer perspective becomes a liability in an agentic context.

Here is why. Suppose you have a route like POST /checkout that simultaneously creates an order, processes payment, sends a confirmation email, updates inventory, and logs an analytics event. For a human clicking a button in a UI, this is fine. For an AI agent, it is a problem. If the payment succeeds but the inventory update fails, you now have a partial success state. The agent cannot easily roll back the payment without knowing exactly what succeeded and what failed. The atomicity that seemed like an engineering virtue has become an agent's debugging nightmare.

The principle that solves this is Single Responsibility at the API route level. Each endpoint should do one clearly defined thing. Complex workflows should be composed by the calling system whether that is your frontend, an orchestration layer, or an AI agent by calling a sequence of single-responsibility operations. Each step is atomic. Each step is independently retryable. Each step returns a clear success or failure that an agent can reason about.

Concretely:

• POST /orders — creates the order record
• POST /payments — processes the payment against an order
• POST /inventory/reserve — reserves stock for a confirmed order
• POST /notifications/send — triggers the confirmation message

If the AI agent calling this sequence encounters a failure at step three, it knows exactly what happened, can retry that step independently, and can roll back cleanly if needed. The intent of each operation is unambiguous, the blast radius of failure is minimal, and the entire workflow is auditable.

This is not a new concept in software engineering it is the Unix philosophy applied to APIs. But it becomes necessary now that AI agents are your users. Definitive, single-responsibility operations are machine-friendly by design.

How to Audit Your AI Readiness Right Now

The good news is that checking how AI-ready your site currently is does not require expensive tooling. Here is a practical audit checklist you can run today:

1. Test your site in the AI assistants themselves: Open ChatGPT, Perplexity, and Claude. Ask questions that your ideal customer would ask questions related to what you sell or do. Does your brand come up? Are you being cited or recommended? This is the ground truth. Do it before you optimise and after, to measure impact. Manually test 10–15 relevant queries across ChatGPT, Perplexity, and Gemini monthly, and document when and how your brand appears.

2. Check your server logs: Look for AI user agents hitting your site. If you see GPTBot, ClaudeBot, PerplexityBot, and similar entries, your site is being crawled by these AI user agents. Check your server logs regularly — if your hosting doesn't offer a UI, server log files are usually located at /var/log/apache2/access.log on Linux-based servers. If you see none, investigate whether your robots.txt is inadvertently blocking them (or whether your Cloudflare or CDN settings are rejecting them).

3. Validate your robots.txt:Make sure you are not accidentally blocking the assistant bots you want to allow. There is a meaningful difference between blocking a training crawler and blocking a real-time assistant crawler, ensure your directives are intentional. A useful starting point: verify AI crawlers are not blocked in your robots.txt and check that your server or CDN is not rejecting AI bot requests, especially if you use Cloudflare.

4. Check whether your llms.txt exists: Visit yourdomain.com/llms.txt. If you get a 404, that is an immediate gap to fill. You need all three files for complete site discoverability: robots.txt for access control, sitemap.xml for traditional search, llms.txt for AI tools. Even a minimal file is better than none.

5. Run Google’s Rich Results Test: Go to search.google.com/test/rich-results and run your key pages through it. This validates your schema markup and tells you what structured data Google (and by extension, AI systems using Google’s index) can detect on your pages.

6. Audit JavaScript rendering: Check whether your important content is server-side rendered or whether it lives behind client-side JavaScript. AI crawlers vary in their ability to execute JavaScript. If your product descriptions, pricing, or key value propositions only appear after JS execution, a crawler that does not render JS will see nothing. Ensure important content is server-side rendered, not hidden behind JavaScript, and confirm content is not locked behind logins, paywalls, or interactive elements.

7. Check AI referral traffic in Google(Your analytics provider)Analytics:. Set up Google Analytics 4 to track traffic from chat.openai.com, perplexity.ai, claude.ai, and gemini.google.com. This is your baseline for measuring GEO impact over time.

The Bigger Picture: Building for the Machine-First Web

What all of this adds up to is a shift in mindset that needs to happen at the engineering level, not just in marketing or SEO teams.

For the past decade, we built for humans, and then tried to make our sites legible to Google’s crawler as a secondary concern. In the AI era, machine readability is a primary product requirement. Search is moving from discovery through lists to discovery through synthesis and generative engines depend on structured signals to know what something means. The same principle that governs how we write documentation for developers, clear structure, explicit intent, predictable contracts, now governs how we build public-facing web products.

Your llms.txt is your product's README for AI systems. Your schema markup is your API contract for machine interpretation. Your single-responsibility API routes are your commitment to agentic consumers who cannot tolerate ambiguity.

This is not extra work on top of building well. This is what building well looks like now.

47% of brands still have no GEO strategy. That is an enormous window right now for the engineers and product teams who move early. The search engine optimisation we have known for decades was always, at its core, about making your product legible to machines so that machines could lead humans to it. The machines have changed. The standards have changed. The question is the same: when someone asks for what you build, does the answer include you?

Build accordingly.

View on Medium