SEO Recap covering October 20, 2026: DeepSeek's Search API and Cloudflare's OpenAI crawl pilot

Built by Stephanie Chung·7 min read·9 stories

Daily summary of what matters in SEO, GEO, AEO, and AI search generated with Claude Code (beware of hallucinations)

Generative Engine Optimization (GEO) & Answer Engine Optimization (AEO)

YouGov Data Shows Audiences Use AI Without Trusting It
Source: Search Engine Journal

  • YouGov found 89% of U.S. adults have at least one concern about using AI, yet 55% still use AI tools weekly, so usage does not signal trust in what AI says about your category.
  • The leading concerns are specific and partly answerable in content:
    • Data privacy and security at 63%
    • Over-reliance on AI at 53%
    • Distrust of AI companies at 48%
    • Doubts about reliability or accuracy at 46%
  • Views on AI split sharply by usage frequency. Weekly users are 52% positive and 22% negative on AI's societal impact, while never-users are 6% positive and 75% negative, a 46-point gap.
  • Segment audience research by AI usage before averaging, and write separate pages for weekly users and never-users. Link primary sources, name authors, date data, and state plainly what happens to user data.
  • Pew's concern figure jumped from 38% in 2022 to 52% in 2023 and has held near 50-52% since.

AI in Search / AI Overviews

Cloudflare and OpenAI Pilot Helps ChatGPT Index Hundreds of Millions of Pages
Source: Search Engine Journal

  • Cloudflare and OpenAI report their research pilot, using ~1% of Cloudflare's network, helped OpenAI discover and index hundreds of millions of new web pages. Results were shared Oct. 15 after the July 8 announcement.
  • Cloudflare's network signals steer OpenAI's crawler toward valuable pages like product manuals, passport requirements, and restaurant menus that few sites link to. Signals also flag pages to skip or keep out of the index, including planted fake login forms.
  • The companies stress a signal does not grant access, and OpenAI independently ranks pages, so discovery through the pilot does not guarantee a citation.
  • OpenAI splits crawling into 2 robots.txt directives. Blocking OAI-SearchBot removes a site from ChatGPT search results, while allowing it but blocking GPTBot keeps the site searchable without feeding model training.
  • Finding new pages can initially increase crawling, and the post gives no figures for crawl volume, server load, citation rates, or referral visits.

Testing DeepSeek's New Search and Fetch API
Source: Search Engine Journal

  • DeepSeek opened a public Search and Fetch API alongside an Oct. 17 Flash model update, returning an authority score per result and claiming to filter SEO spam, GEO manipulation, and low-quality AI content. It indexes hundreds of billions of pages and handles hundreds of millions of searches a day.
  • Across 1,000 results, rerank scores ran 0.75-2.70 with a median of 1.63, and 68% scored 1.5 or above. Site probes showed every URL on a host carried the same authority tier, so the documented page authority behaved like a host score.
  • Spam filtering let questionable material through. Spam PDFs on government and university hosts scored the top authority tier of 4, and self-published Zenodo frameworks ranked #1 and #3 for "generative engine optimization."
  • The Fetch API presented as Firefox from Hong Kong on Huawei Cloud IPs, ran no JavaScript, carried no DeepSeek identifier, and returned content from URLs disallowed by robots.txt.
  • Practical testing notes for SEOs:
    • Set localization explicitly since it defaults to the caller's IP and reshuffles results
    • Date filters removed every undated page (11% of results had no date), so add machine-readable dates
    • Package registries and app stores (PyPI, npm, Apify, Chrome Web Store, WordPress.org, GitHub) outranked original tool content

Personal Shopping Agents Like Meta's Muse and OpenAI's Dots Split the Web
Source: Search Engine Journal

  • Muse and Dots carry out web tasks for a person, researching products, filling forms, and completing checkout. Muse is free for most uses to anyone 18 or over in the U.S. and Canada, and The Information reports more than 3 million people prompt Muse weekly.
  • Businesses reach these agents through one of 2 tiers:
    • A connector that integrates via API (Muse launched with dozens of partners plus the full Shopify catalog, Dots connects to over 4,000 apps)
    • A cloud Chromium browser that visits unenrolled sites disguised as the customer
  • Neither Meta nor OpenAI publishes a user agent, header, or signature for the agent browsers, so there is no name to put in robots.txt. Amazon blocked Muse via its Conditions of Use instead, and HUMAN Security says its product can detect Muse.
  • Muse reads pages through the accessibility tree, not the raw DOM, and runs no JavaScript of its own. On a tested Shopify product page, 6 sold-out sizes showed as crossed out to shoppers but appeared as 7 identical available buttons in the accessibility tree.
  • Agent purchases through a connector skip your site analytics entirely, and browser-tier visits will likely be counted as human customers. llms.txt and markdown versions are useless to an agent that came to use buttons, forms, and checkout.

Google Drops AI Overviews Label on Some Desktop Results
Source: Search Engine Roundtable

  • Google appears to be removing the AI Overview label from some desktop results, while the same feature on mobile still shows the label. Glenn Gabe spotted it on AI Overviews for shows and movies.
  • Barry Schwartz suspects a bug and has reached out to Google, noting Google has shipped missing AI labels as bugs before.
  • The missing labels may explain Semrush reporting a drop in how often AI Overviews appear in Google Search.

Technical SEO

Mueller: Square Brackets in URLs Are Not Equivalent to Their Encoded Form
Source: Search Engine Roundtable

  • John Mueller confirmed that "?filter[]=123" and "?filter%5B%5D=123" are not equivalent, because under RFC 3986 brackets and plus signs are reserved characters protected from normalization.
  • URLs using raw brackets are technically invalid since [ ] are only valid in hostnames wrapping IPv6 addresses, though browsers hex-encode them and messy sites still index fine.
  • Pick one URL form and use it consistently across links, canonical tags, sitemaps, and structured data. Google de-dupes versions serving identical content, but Mueller would not build a new site on the invalid form.
  • Mueller warned other crawlers, indexers, and fetchers may handle these characters less precisely, so double-check the ones you care about. The same care applies to spaces, plus signs, and %20, which vary by path versus query string.

Organic Search & Algorithm Updates

Chartbeat: Search Share of Publisher Page Views Fell in 9 of 10 Regions
Source: Search Engine Journal

  • Search engines drove 12% of page views at North American publishers in Chartbeat's Q3 2026 Audience Report, down from 21% a year earlier.
  • 9 of 10 regions saw search share decline year over year. Central Asia dropped most, from 53% to 33%, while Africa was the only region to rise, from 41% to 46%.
  • North American share slid from 21% to 16% in Q4 2025, then 13% in Q2 2026 and 12% in Q3, with 5 of the 9 points lost between Q3 and Q4 2025.
  • Earlier Chartbeat reports attributed the decline to AI's growing impact on search and content discovery. The latest report does not state a cause, counts all search engines as one category, and does not disclose how many sites were measured.

Bing Tests Hiding Sitelinks Under Top Result Snippets
Source: Search Engine Roundtable

  • Microsoft appears to be testing Bing search results without sitelinks under the top result snippet, first spotted by Khushal Bherwani and replicated by Barry Schwartz.
  • The same query showed sitelinks in one browser and none in another, indicating an A/B test rather than a full rollout.

Google Tests Removing the Horizontal Rule Under Search Tabs
Source: Search Engine Roundtable

  • Google is testing removal of the horizontal rule line beneath the search tabs below the search box, spotted by Sachin Patel and replicated by Barry Schwartz.
  • The change is minor cosmetic SERP chrome, affecting the thematic break rendered between the header and results.
Published on
geoai searchtechnical seoorganic searchdigest