When AI Retrieves the Wrong Page

This morning we asked an AI system to inspect several pages on Wayli.

The pages were live.

They returned successful HTTP responses.

They contained page-specific titles, meaningful HTML, canonical URLs and structured data.

But the AI did not always retrieve them.

When given the homepage, it did not immediately find the homepage.

When given some deeper pages, it returned the homepage instead.

For other pages, it surfaced unrelated competitor websites.

The problem was not immediately obvious because the AI still produced an answer.

It had simply answered from the wrong evidence.

What happened

We tested several Wayli URLs.

What the AI retrieved

🏠 Homepage wayli.uk/ → ✅ Correct page retrieved

🤖 AI Readiness wayli.uk/ai-readiness → ⚠️ Homepage substituted

🏡 Mortgage decision tool wayli.uk/mortgage-decision → ❌ Unrelated mortgage providers returned

🌅 Retirement planner wayli.uk/retirement-planner → ❌ Unrelated retirement planners returned

🎓 Student loan calculator wayli.uk/student-loan-calculator → ❌ A competitor’s calculator returned

Only one of the five requested pages was retrieved correctly.

The requested pages themselves appeared healthy.

Previous checks had confirmed that the main decision tools returned successful responses with individual page titles, real page content, input fields and server-delivered HTML.

Checks with a second AI model could see everything as we expected

In an earlier review, another AI system eventually acknowledged that its own fetch tool was the unreliable part of the test rather than the website. It revised its score after being shown direct response evidence from the site.

That matters because the first review sounded authoritative.

It was also based on the wrong evidence.

A possible explanation

It is tempting to imagine that when a user gives an AI a URL, the AI simply opens that URL.

That may not always be what happens.

A retrieval system may instead:

Receive the user’s question
        ↓
Search an internal or external index
        ↓
Choose what it believes is the best source
        ↓
Pass that source to the language model
        ↓
Generate an answer

If the intended page is not selected, the language model may never see it.

The model can then reason perfectly well over completely irrelevant evidence.

The resulting answer may be fluent, detailed and wrong.

Our tests suggest this is more consistent with a retrieval or indexing limitation than a problem with Wayli’s sitemap or URL structure. The sitemap contained canonical, readable URLs with logical groupings and no obvious duplication.

Retrieval is not understanding

AI Readiness has so far concentrated heavily on whether a website presents enough evidence for AI to understand it.

This test revealed an earlier question.

Did the AI retrieve the right evidence in the first place?

That suggests several distinct stages:

Crawlability
      ↓
Indexing
      ↓
Retrieval
      ↓
Understanding
      ↓
Reasoning
      ↓
Answer

These stages should not be treated as interchangeable.

A page may be crawlable but not indexed.

It may be indexed but not selected.

It may be selected but misunderstood.

It may be understood correctly but used in weak reasoning.

Each is a different failure.

Why the distinction matters

Without this distinction, a business may receive an AI review saying:

Your website does not explain its products clearly.

But the AI may not have reviewed the product page at all.

It may have seen only the homepage.

Or an old indexed result.

Or a competitor.

Improving the wording on the product page would not resolve that failure because the product page was never part of the evidence.

This is why AI assessments need to distinguish between:

• evidence that exists,
• evidence that can be crawled,
• evidence that has been indexed,
• evidence that was actually retrieved,
• and conclusions drawn from that evidence.

What we can verify

The current evidence supports several conclusions.

The Wayli pages tested were publicly available and returned valid responses.

The sitemap appeared structurally healthy.

The retrieval result varied depending on the page requested.

The AI sometimes substituted the homepage or an unrelated website.

A separate review later accepted direct response evidence showing that the site, rather than its pages, was not the source of the earlier failure.

What we cannot yet verify is why the retrieval layer made those substitutions.

Possible explanations include:

• a deep page had not yet entered the retrieval provider’s index,
• the homepage had stronger ranking signals,
• the supplied URL was treated as a search hint rather than a direct fetch instruction,
• the retriever preferred a more established competitor for a generic topic,
• or the retrieval tool itself behaved inconsistently.

These remain hypotheses.

What not to do

The immediate response from another AI was to recommend SSR, an llms.txt file, crawler directives and additional structured data.

Those may be sensible checks.

But they should not be treated as the answer before the failure has been identified.

Wayli had already implemented much of that foundation.

Adding more technical signals cannot guarantee that a proprietary retrieval system will select the intended page.

The first task is therefore not optimisation.

It is diagnosis.

A new measurement for AI Readiness

This incident suggests that retrieval quality should become an explicit part of the assessment methodology.

Possible tests include:

| Test | Result | | ------------------------------------------------------------ | -------- | | Can the homepage be retrieved? | Yes / No | | Can important deep pages be retrieved? | Yes / No | | Was the requested page selected? | Yes / No | | Was the homepage substituted? | Yes / No | | Was a competitor substituted? | Yes / No | | Was enough relevant evidence retrieved to support reasoning? | Yes / No |

A report should then be able to say:

Crawlability: Passed

Indexing: Unconfirmed

Retrieval: Failed
The requested product page was not selected.

Understanding: Not assessed

Reasoning: Not assessed

That is far more useful than presenting a confident verdict about content the AI never saw.

The wider Wayli lesson

Wayli began with a simple idea.

AI can only reason from the information available.

This test adds an important qualification.

The information may exist.

It may be well written.

It may be technically accessible.

But it still has to reach the model.

The quality of an AI answer therefore depends not only on the evidence a business publishes, but also on the evidence the retrieval system chooses to provide.

Before judging the answer, we should ask a more basic question.

Did AI answer from the right page?