SEO Recap covering July 25, 2026: AI crawler tolls and Google's Q2 click claims

Built by Stephanie Chung·5 min read·4 stories

Daily summary of what matters in SEO, GEO, AEO, and AI search generated with Claude Code (beware of hallucinations)

Generative Engine Optimization (GEO) & Answer Engine Optimization (AEO)

HTTP 402 (Payment Required), dormant since 1997, is now being used to charge AI crawlers: Cloudflare launched pay-per-crawl on July 1, 2025, and AWS added a 402 capability to its Web Application Firewall on June 15, 2026 via the x402 protocol with stablecoin settlement through Coinbase.
Cloudflare's data attributes nearly 80% of AI bot activity to training, with search-purpose fetches that can return a citation being a small slice, so most crawling extracts content and sends no referral back.
The author frames tolling as a visibility tradeoff, not a revenue line: the crawler you charge and the AI answer that refers customers are often the same pipeline, and Adobe's 2026 data put AI-referred traffic to US retailers up 393% year over year.
The recommended test before charging: meter one bot on one path, watch its effect on both crawl volume and your presence in AI answers, then set policy based on that data rather than launch hype.
The unsettled question is whether anonymous bots will pay: x402 aims for payment without prior registration, which would extend tolls beyond named crawlers to the anonymous majority of automated traffic.

Google and DeepMind's AI & Economy ATLAS report covers 14.65 million interactions from the Gemini app, AI Mode in Search, and the Gemini API, comparing US non-work conversations against the American Time Use Survey (April 6 to 19, 2026).
Subjects people bring to AI far more than they spend time on them:
- Government services and civic obligations, almost 20 times
- Eating and drinking topics, about 18 times (though people eat far more than they ask)
- Professional and personal care services, more than 7 times
- Education, 5.8 times
- Consumer purchases, nearly 3 times
TV and movies, washing and dressing, cleaning, and cooking sit at the bottom, coming up far less in AI conversations than the time people spend on them.
The report covers conversations inside Google's own products with no click data, so it cannot show whether high-friction health, legal, money, and government queries send any traffic to websites. About half of those came in outside working hours.

John Mueller diagnosed a Reddit case where a Shopify store's search box generated spammy URLs that Google indexed despite a /search disallow in robots.txt.
The cause was a robots.txt structure with both a Googlebot-specific section and a user-agent: * section. More specific rules win, so Googlebot only obeyed its own block and ignored the shared rules, which had to be copied into the Googlebot section or the user-agents grouped together.
Mueller's key point: robots.txt controls crawling, not indexing, so a noindex directive is the effective fix for search results pages.
Mitigation by platform:
- Shopify: add a noindex meta tag to the search template in theme.liquid, and do not block search pages in robots.txt or the noindex can't be read
- WordPress: Yoast, Rank Math, and AIOSEO set search results pages to noindex by default

Alphabet's Q2 Search & Other revenue rose 17% year over year to $63.27 billion, a slowdown from 19% in Q1 2026 after four quarters of accelerating growth, but the filings still provide no count of clicks sent to websites.
The piece groups Google's public traffic statements into 3 untestable categories:
- Usage claims (queries at an all-time high, record usage during a World Cup match)
- Volume claims (AI features send billions of clicks weekly, billions daily) with no exact figures
- Quality claims (Liz Reid says AI Overviews mostly remove bounce clicks, organic clicks relatively stable)
Outside measurements each test only a slice: Seer Interactive found organic CTR on AI Overview queries fell from 1.76% in 2024 to 0.61% in 2025, and Pew found users clicked 8% of the time with AI Overviews versus 15% without.
Similarweb found brands recommended in ChatGPT answers were 2.5 times more likely to get a site visit within a week, with 55.9% of that traffic coming from branded searches, an indirect route Google's own click claims do not address.
The author's takeaway: Search revenue can rise without revealing whether organic visits grew, and no public dataset can reconcile Google's aggregate claims, so your own site data is the closest thing to a checkable number.

Published on July 26, 2026