Skip to content
TrustList
Blog

From ten blue links to answer engines: how the role of search engines is changing

Editorial

By TrustList Editorial

AI crawlers read our site ten times more than search engines did in one night. How answer engines, the US Google remedies, the EU DMA fine and the Bing API retirement change search.

About From ten blue links to answer engines: how the role of search engines is changing

From ten blue links to answer engines: how the role of search engines is changing

In one overnight window of 9.3 hours, from 21:19 UTC on 24 September to 06:39 UTC on 25 September 2026, the traditional web search engines together fetched 2,703 pages from trustlist.uk. Googlebot took 221 of them, Bingbot 2,180, Baidu 200, DuckDuckBot 65 and Yandex 37. In the same window, crawlers and fetchers run by AI companies took 27,591 pages, ten times as many. OpenAI's GPTBot alone took 15,285.

That is one night on one site, a UK directory of businesses and software products, and it is a sample rather than a month. But it captures something that is hard to see from the outside. The organisations reading the web most heavily are no longer only the ones that send you visitors in return. The job a search engine used to do, reading everything and pointing people to the best pages, is being split between several kinds of institution, and the courts and regulators are changing the rules for the one that held it longest.

This article looks at search engines as institutions rather than as a marketing channel. It covers the shift from lists of links to written answers, the US court remedies against Google, the EU's first Digital Markets Act fine on Google Search, Microsoft's decision to stop selling its search results to developers, and what all this means for a business buyer researching suppliers and for a supplier who wants to be found. For the practitioner's view, read is SEO dying or changing. For our own traffic story, read what 657 clicks taught us.

Who reads the web now

For most of the last twenty years a site owner could think of the crawlers visiting their pages in simple terms. A search engine crawler read the page, the page went into an index, and the engine sent visitors when a search matched. The bargain was reading in exchange for referrals.

Our log sample shows how much that has changed. Of 77,221 HTML page requests in the window:

Group Page requests Share
Browser-like user agents 35,401 45.8%
AI company crawlers and fetchers 27,591 35.7%
Other unnamed bots and scripts 8,793 11.4%
Traditional search engine crawlers 2,703 3.5%
SEO tools (Moz, Majestic, Semrush) 2,732 3.5%

The browser-like row overstates human visits, because our analytics show much of it is an automated browser farm. Within it, 27 page views came from a Google search results page. We allow AI crawlers and SEO tools in as a matter of policy.

The AI group is not one thing. OpenAI documents three separate agents. GPTBot collects content for training its models. OAI-SearchBot is used to surface websites in ChatGPT's search answers, and OpenAI says sites that opt out of it will not be shown there. ChatGPT-User makes fetches on behalf of a person during a conversation, and OpenAI notes that robots.txt rules may not apply to those because a user started them. In our window the three made 15,285, 5,359 and 66 requests respectively. Meta's crawlers made 4,519, Applebot 1,747 and Anthropic's ClaudeBot 614.

Cloudflare, which carries a large share of web traffic, has measured the same imbalance across its network. In figures published in August 2025 it reported that in July 2025 OpenAI crawled about 1,091 pages for every visitor it referred, Anthropic about 38,066, and Google a single-digit number. It also reported that training accounted for nearly 80% of AI crawler activity.

The old bargain has not disappeared. Google still reads the web and still sends people to it. But the reading and the referring are no longer done by the same organisations in the same proportions.

From lists of links to written answers

The second change is in what a search engine shows.

Google shows AI Overviews, written summaries above the links. By 20 May 2025 it said they were available in more than 200 countries and territories and more than 40 languages, and that in large markets such as the US and India they were driving an increase of over 10% in Google use for the kinds of queries that show them. AI Mode, a conversational search interface, launched in the UK on 28 July 2025. Google described it as breaking a question into subtopics and running many searches at once, a technique it calls query fan-out, and said it kept "prominent links for people to click on".

By its second-quarter 2026 results on 22 July 2026, Alphabet said AI Mode had passed 1 billion monthly active users since its global expansion the previous October, and that AI Overviews and AI Mode had been brought together into one Search experience. Sundar Pichai said Google was "sending billions of clicks to websites every week through AI features in Search". Google Search and other revenue for the quarter was $63.271 billion, against $54.190 billion a year earlier.

What this does to clicks has been measured on samples. Pew Research Center tracked the browsing of 900 US adults in March 2025. When a Google search showed an AI summary, users clicked a traditional result in 8% of visits, against 15% when there was none, and clicked a link inside the summary in 1%. Google's position, set out by its head of Search Liz Reid in August 2025, is that total organic clicks to websites have been "relatively stable year-over-year" but are shifting between sites.

Both can be true. The engine is answering more of the question itself, and the clicks that remain are concentrated on fewer, more specific destinations. In institutional terms, the search engine is moving from directory to publisher: it now writes the first answer, and chooses which sources to credit.

The courts: what the US search remedies ordered

The third change is legal. In United States v. Google, the US Department of Justice and a group of states argued that Google had unlawfully maintained a monopoly in general search. The court found against Google on liability, and on 2 September 2025 it issued its remedies ruling.

According to the Justice Department's announcement that day, the court:

  • barred Google from entering or maintaining exclusive contracts relating to the distribution of Google Search, Chrome, Google Assistant and the Gemini app;
  • barred Google from conditioning payments on keeping those products on a device, browser or search access point for more than one year;
  • required Google to make certain search index and user-interaction data available to certain competitors;
  • required Google to offer certain competitors search and search text ads syndication services.

The court declined to order Google to sell Chrome or Android, and did not ban all payments to distribution partners. The Department's case page lists the court's memorandum opinion and final judgment dated 5 December 2025. It also lists status reports on Google's compliance filed on 4 May 2026 and 17 September 2026, and the government's appellate brief of 28 July 2026, which answers Google's appeal and opens the government's own cross-appeal. The case is not over.

Two points matter beyond the legal detail. First, the remedies reach AI products. The Gemini app is named alongside Search and Chrome, and the Department said the ruling recognised the need to stop Google using the same tactics for its generative AI products. The court treated answer engines as part of the same contest as search. Second, the remedy is about access to the index and to user data. The court's view was that the thing that makes a search engine hard to challenge is the index and the record of what people click, and it ordered some of that to be shared.

The Justice Department also won a separate case about Google's advertising technology. On 16 September 2026 it announced remedies in that case, including required integrations with Prebid and a monitor and technical committee for six years. That case concerns the tools that sell display ads on publishers' sites rather than search itself, but it affects the same publishers who depend on search traffic.

Europe: the first DMA fine on Google Search

The European Union took a different route. The Digital Markets Act sets obligations in advance for large "gatekeeper" platforms instead of waiting for a monopoly case. One of them forbids a gatekeeper from treating its own services more favourably in ranking than similar services from third parties.

On 23 July 2026 the European Commission fined Google €460 million for breaching that obligation in Google Search. The Commission found that Google gave preferential treatment to its own services, "including shopping, hotels, transport and sports results", by showing them more prominently, including at the top of the results page or with enhanced visuals and filters. In the same announcement it fined Google €430 million for a separate breach concerning Google Play, a total of €890 million.

For a business listed by or competing with a vertical search service, this is the most direct legal statement yet that the layout of a results page is a regulated matter in the EU, not just a product decision.

The index as infrastructure

The fourth change is quieter. On 11 August 2025 Microsoft retired the Bing Search APIs, which for years let developers and smaller search services buy Bing's results and show them in their own products. Microsoft's lifecycle notice said existing instances would be "decommissioned completely" and pointed customers to Grounding with Bing Search, part of Azure AI Agents, which lets an AI agent use real-time web data when writing its responses.

Put that beside the US remedy and a pattern appears. A web-scale index has become infrastructure. One company has been ordered by a court to share parts of its index with qualified rivals. The other has stopped selling its index as a list of results and now sells it as grounding for AI agents. Either way, the index is increasingly consumed by machines that write answers, not by people reading a list.

That is consistent with what we see in our logs. The crawlers of companies that run answer engines are reading our pages far more often than the crawler of the engine that sends us most of our search visitors.

What this means for a B2B buyer researching suppliers

If you are researching software or a service provider, the tools you use to search are changing in ways that affect what you see.

The first answer is now written for you. An AI summary or answer engine will often name suppliers before you see any list of links. It chose those names from sources it trusts, and it may not show you why. Ask it for its sources, open them, and check whether they are primary: the supplier's own site, a regulator's register, a filing, or a directory that states how it counted.

Answers can be stale. Our own catalogue shows how quickly facts decay. In a random sample of 1,000 company listings on 24 September 2026, 171 websites no longer led to the business as listed. Of 5,092 software listings in our catalogue that show 2020 prices with no date, a sample of 30 checked against the vendors' pages found only 6 still matched. An answer built from an old page will repeat old facts confidently.

Vertical results may favour the engine's own services. The EU found exactly that for shopping, hotels, transport and sports. When a results page shows you a box of suppliers, ask who built the box.

Your research is being read too. When you ask an answer engine a question, a fetcher like ChatGPT-User may retrieve pages live to answer it. That is useful, but it also means the answer depends on which sites allow such fetchers.

What this means for a supplier who wants to be found

If you sell to businesses, the question is no longer only "do we rank?" but "are we the source an answer depends on?"

Be the primary source about yourself. Our Search Console data for the 92 days to 18 September 2026 showed that pages about one named business or product earned 528 of our 657 clicks, at a 1.6% click-through rate, while broad ranking pages earned 67 from about 255,000 impressions. Searches for a specific company still lead to pages about that company. Make sure your own site states plainly what you do, who you serve, where you are, what you charge if you publish prices, and when that was last true.

Decide which crawlers to admit, and why. Blocking GPTBot keeps your content out of OpenAI's training; blocking OAI-SearchBot keeps you out of ChatGPT's search answers. They are separate decisions. Google says its AI features need nothing beyond normal indexing and snippet eligibility, so a page excluded from Google's index is excluded from its AI answers too.

Publish things that must be cited. An answer engine can summarise a generic description of your service without naming you. It cannot state a figure you counted without taking it from you. Case numbers, dated prices, notice periods, response times with a method: these are what survive summarisation.

Keep your listings current. Directories, including ours, and answer engines draw on what is published about you. A dead website, an old address or an undated price is repeated until someone corrects it.

A checklist for both sides

For buyers:

  1. Ask every AI answer for its sources, and open at least two.
  2. Prefer sources that state a date and how they counted.
  3. Check the supplier's own website is live and matches what the answer says.
  4. Treat any price without a date as unknown.
  5. Notice when a results box comes from the engine's own service.

For suppliers:

  1. Search for your company by name in two engines and one answer engine, and record what each says.
  2. Correct any directory listing, including your TrustList listing, that is wrong or out of date.
  3. Put a dated, plain statement of services, location and prices on your own site.
  4. Review robots.txt against the documented crawler names, one decision per crawler.
  5. Publish one counted, dated fact about your work each quarter.
  6. Check your server logs once a quarter to see who is reading you.

Search engines are not disappearing. They are becoming answer writers, index suppliers and regulated gatekeepers at the same time, and new institutions are doing part of their old job. The businesses that do well will be the ones those institutions have to cite.

Sources

Keep exploring

More on TrustList

Everything here links back to the same verified catalogue. Pick your next stop.