Meta pauses the terms that would let its crawlers index business sites

Meta pauses the terms that would let its crawlers index business sites

20 October 2026: Meta's updated Commercial Terms, which apply to any business that runs ads, sells products, manages a Page or uses Meta Business Suite, authorise Meta to crawl, index, cache and analyse the websites tied to that business and to pull updates from them periodically. Meta first set the update to take effect on 30 October 2026, then said on 23 October that it would move the date to 20 November. A business accepts the terms by continuing to use Meta's products for commercial purposes.

Not yet independently verified. Meta's pause is reported by Reuters from a company spokesperson on 6 November 2026, and Meta's terms pages refused our requests. The clause quoted here is from the preview of the paused update, and Meta has given no new date. We will update this when it can be confirmed, and remove this note.

Update, 9 November 2026: The update will not take effect on 20 November. Reuters reported on 6 November that Meta has paused the planned changes to its Commercial Terms, Product Catalog Terms, Self-Serve Ad Terms and Ad Creative Generative AI Terms. "We're pausing the changes as we continue to engage with stakeholders and take our time to get this right. In the meantime, our existing terms remain in effect," a spokesperson said. No new date has been given, so the clause described below is not in force. The crawler controls in the last section do not depend on the terms.

What the new clause allows

The section on licences now carries a paragraph that did not exist in the current terms. A business authorises Meta, or anyone acting for it, to "use automated software (such as web scrapers, bots, robots, spiders, crawlers and user agents) and create test credentials" to access the domains, pages and URLs associated with its products or services, or made available through anyone's use of Meta's products. The stated purposes are to "provide, develop, improve, research and maintain advertising and business products", including tools that create and deliver ads, tell users about a business's products and connect it with interested users, and to protect the integrity and security of Meta's products.

Two parts deserve a careful read. "Test credentials" means Meta's tools may create accounts to reach pages behind a login, so a shop or booking site with self-service sign-up should expect them. And the scope covers URLs "made available through your or users' use" of Meta's products, which reaches any link a customer shares, not only the site in the business's own profile.

The same update adds an AI features section: the business takes on the risk of any content Meta's AI tools create or change for it and must not strip watermarks or provenance metadata from AI-generated content.

Meta narrowed the wording after the first reports

Nieman Lab first reported the change, and SmartCompany reported on 20 October that Meta adjusted the forthcoming text afterwards, tying the crawling to advertising and business products. A Meta spokesperson told SmartCompany the clause consolidates several agreements and "does not change how Meta uses web crawlers". The terms themselves leave no negotiation: "If you do not agree to the updated terms, you may stop all access or use of our Products for business or commercial purposes."

Which crawler to block

Meta's crawler documentation names separate user agents. Meta-ExternalAds crawls "for use cases such as improving advertising and other business-related products and services", Meta-ExternalAgent crawls for uses that include "training foundation AI models", and Meta-WebIndexer helps Meta AI cite and link content. Each honours a disallow rule in robots.txt. Two do not always: FacebookExternalHit, which builds link previews, may bypass robots.txt for security checks, and Meta-ExternalFetcher, which fetches links at a user's request, "may bypass robots.txt rules". Blocking the ads crawler can also degrade how Meta's ad tools read a product catalogue, so marketing and web teams should decide together. Our social media marketing software category lists 14 products and our social media management category 12.

Update, 24 October 2026: Meta has pushed the effective date back to 20 November 2026. Reuters reported on 23 October that the move followed complaints by publishers and digital advertisers to the EU and UK competition authorities, which Meta rejects. A Meta spokesperson said: "We are extending the effective date of our updated Commercial Terms and Self-Serve Ad Terms to 20 November to allow more time for us to explain to our partners and stakeholders what's changing and what's not."

Related on TrustList:

• CMA opens abuse-of-dominance case over Meta's terms for publisher content

Sources

• Meta: Commercial Terms, updated version effective 30 October 2026 (read 2026-10-20), 20 October 2026
• Meta for Developers: Meta web crawlers (read 2026-10-20), 20 October 2026
• SmartCompany: Meta's new AI scraping terms put small businesses on notice, 20 October 2026
• The Hindu (Reuters): Meta to delay new commercial terms criticised by publishers, advertisers, 23 October 2026
• Reuters via The Star: Meta pauses planned terms updates, keeps existing rules in place, 6 November 2026