Free site checks: audit, citations, AI crawlers

Three checks that call no model and cost nothing, the site audit of what AI crawlers can read, the citation check of which cited pages name you, and AI crawler counts from your access logs.

Can AI assistants read my site? popcorn audit checks what an AI crawler sees on each of your sites: whether robots.txt or a firewall blocks it, whether there’s an llms.txt and a fresh sitemap, and whether each page has its content in the HTML or is empty without JavaScript. Two more checks sit beside it: the citation check, of which pages the answers cite name you and which name only competitors, and AI crawler counts from your own access logs.

None of the three calls a model. They’re plain HTTP requests and your own logs, so they cost nothing, on any setup, and don’t count against the daily budget.

Site audit

popcorn audit                 # every site in [[brand.sites]] (and brand.domains)
popcorn audit example.com     # just one, which needn't be yours (not kept)
popcorn audit -pages 20 -info # more sitemap pages; list info findings too
popcorn audit -json           # the findings as JSON

Also Site audit in the web UI (/audit, “Audit now”), POST /api/audit to run one and GET /api/audit for the latest, and the MCP tool audit_sites.

For each site it fetches robots.txt, the home page, /llms.txt, the sitemap and up to 10 pages from the sitemap, spread through it. It fetches as Mozilla/5.0 (compatible; Popcorn-Audit/<version>; +https://popcorn.zone), 4 requests at a time, with a 15-second timeout and a 2 MB cap per page, and reads the raw HTML, as a crawler that runs no JavaScript does.

• A host/path site (github.com/example) gets its robots.txt rules checked for that path, but no llms.txt or sitemap check: those belong to the host.
• Sites of kind code or community don’t get their markup checked. It isn’t yours to fix.
• The newest audit of your own sites is kept. If it’s less than 30 days old, its errors and warnings go to the analyst with the next fix list, so a fix can say “this page is empty without JavaScript” rather than only “this page wasn’t cited”.

Each finding has a severity, a check id, what it found and a fix.

Crawling rules

The firewall check only sends each bot’s user agent, from Popcorn’s address. A firewall that lets verified bots through by IP may block this check and still admit the real crawler. Confirm in the firewall’s log, or with the crawler counts below, before changing rules.

llms.txt and the sitemap

Pages

A site listed in a form Popcorn can’t read gets site.invalid (error): list it as a host (example.com) or a host and path (github.com/example).

Citation check

popcorn check-citations <run-id>         # the 25 most-cited third-party pages
popcorn check-citations -n 50 <run-id>

Also on the Site audit page (“Check citations”, for any finished run), POST /api/runs/{id}/citation-check (?n=, default 25) to run one and GET for the run’s latest, and the MCP tool check_citations.

It takes the run’s third-party pages, most cited first, then most surfaced in searches, and fetches each one as Popcorn, where its robots.txt allows Popcorn-Audit. Each page is fetched at most once a day and kept for a month. It reads the page’s visible text and says whether it names your brand or an alias, and which of the set’s competitors.

A page that names competitors but not you is marked outreach: the answers read it, and it leaves you out. That’s the same idea as the run page’s Cited without your brand list, checked against the page itself rather than the answers.

Matching is by whole word, ignoring case. That’s stricter than the grader’s own check, so a short alias doesn’t match inside another word.

AI crawlers

popcorn crawl import access.log access.log.1.gz        # nginx, Apache
popcorn crawl import -host docs.example.com docs.log   # the site, for lines that don't record one
popcorn crawl import cloudflare-*.json caddy.json -    # - reads stdin
popcorn crawl report -days 30

Also AI crawlers in the web UI (/crawlers), GET /api/crawlers?days=30, and the MCP tool get_crawlers.

It reads:

• common and combined log format, the nginx and Apache defaults, with or without a leading virtual host;
• Caddy’s JSON access log;
• Cloudflare Logpush JSON (ClientRequestUserAgent, ClientRequestURI, ClientRequestHost, EdgeStartTimestamp, ClientIP, EdgeResponseStatus, ClientRequestReferer).

Lines with no host (common and combined) count under -host, or your first site. Requests for CSS, scripts, images and fonts are skipped. Gzipped logs are read as they are.

Each request is classed by its user agent:

Unverified: user agent only. Anyone can send a crawler’s user agent, and nothing here checks the request’s IP against the operator’s published ranges. Read the counts as what the logs claim.

Popcorn keeps only daily counts per host, page, class and bot, not log lines, plus a hash of each counted request, so importing the same log twice, or one that overlaps it, counts nothing twice.

The page lists each of your pages’ fetches by class, and AI referrals, beside how often the latest finished run of each set cited and surfaced it, and flags:

• crawled, never cited: AI crawlers fetched it, but no answer in the latest runs cited it;
• cited, never crawled by search bots: answers cite it, but no assistant or AI search crawler fetched it in the window. Only for hosts your logs cover.

Where to start

Run popcorn audit before your first paid run. It’s free, it takes a minute, and an error there, such as a firewall that blocks ChatGPT-User or a docs site that’s empty without JavaScript, explains a lot of “never cited” before you’ve asked a single question. Then read Reading results for what a run adds.