ATAEO Tracker

AEOTrackerBot

AEOTrackerBot is the web reader of AEO Tracker, a service that checks how AI answer engines describe financial advisory firms. It reads a small number of public pages from a firm's own website when someone starts an analysis of that firm in the product. It is not a continuous crawler and it does not index the web.

How it identifies itself

Every request carries this User-Agent header:

AEOTrackerBot/0.1 (+https://aeo-tracker.com/bot; contact: bot@aeo-tracker.com)

The product token is AEOTrackerBot. Headers are never spoofed and the reader never presents itself as a browser or as another bot.

Who operates it

AEO Tracker is the name under which this reader and the AI Visibility product are operated. The legal entity behind it will be named here once it is formed. Requests come from Amazon Web Services in the us-east-1 region; there is no fixed address range to allow-list, which is why the User-Agent header and this page are the identification.

What it reads

  • Only pages on the firm's own host. It does not follow links to other sites.
  • At most 8 pages per analysis, typically the home, about, team, services and disclosures pages.
  • At most one request per second per host (1000 ms between requests).
  • Each page read stops after 15 seconds and a whole visit after 150 seconds. At most 3 redirects are followed for any one page, and only within the same site.
  • Public addresses only. Private, loopback and internal addresses are never contacted, redirects included.

What stops it

  • robots.txt is fetched first, following up to 5 redirects between the bare and www. forms of the site. Rules for AEOTrackerBot or * are followed; if the root is disallowed, nothing is read. If robots.txt cannot be read (a server error, no answer, or a redirect elsewhere), nothing is read either.
  • A noindex directive on the home page, in an X-Robots-Tag header or a robots meta tag, stops reading beyond that page.
  • Terms-of-use language about automated access stops reading beyond the home and terms pages and is shown to the person who started the analysis.
  • A 401, 403 or 429 response, a login wall, a CAPTCHA or a bot challenge on any page ends the visit. Nothing is bypassed and nothing is retried: a Retry-After is respected by not asking again.

What is kept

A copy of each fetched page and a log of every request decision (address, rule applied, outcome, reason) are stored as evidence for that analysis, so the person who ran it can see where each statement in the draft firm profile came from. Text extracted from the fetched pages is sent to OpenAI's API to draft that profile. Copies are stored on Amazon Web Services in the us-east-1 region. The retention period for fetched pages has not been set yet; this page will be updated when it is.

Opting out

Add this to the site's robots.txt. It takes effect on the next analysis of that site:

User-agent: AEOTrackerBot
Disallow: /

Or write to bot@aeo-tracker.com and the site is added to a do-not-read list the reader checks before every analysis. The mailbox is read by a person, not automatically; please allow a few days.

Visiting this page

The web server keeps standard access logs (IP address, browser, pages requested) for 90 days for security and troubleshooting. This page sets no cookies and loads no analytics.