# The account pages are NOT disallowed here, on purpose.
#
# They ship `<meta name="robots" content="noindex, follow">`, and a crawler that
# is forbidden to fetch a page cannot read the noindex it contains. Blocking and
# de-indexing are opposite instructions: Disallow keeps the content unseen but
# leaves the URL eligible for the index, so the page can still surface with no
# description. To keep something out of the index, let it be crawled and let the
# noindex do the work.
#
# The previous rules also did not do what they looked like: with
# `trailingSlash: "never"` every account URL is `/{lang}/account…` with no
# trailing slash, so `Disallow: /*/account/` missed `/en/account` entirely while
# matching `/en/account/login` — blocking exactly the pages whose noindex
# mattered, and none of the ones it was aimed at. It forbade 45 noindex pages
# across the 15 locales.
#
# /api/ stays disallowed: it is served by the Worker, returns no HTML, and
# therefore has no meta tag to carry a directive.
User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://aistrategyminer.com/sitemap-index.xml