Archive / Find out who’s visiting your site: people, crawlers, or both

Find out who’s visiting your site: people, crawlers, or both

Compare aggregate host and analytics data without guessing how much traffic is human.

For: Solo founders, Small business owners

Topics: Websites, Marketing

If the host counts 1,200 requests and analytics counts 170 sessions, subtracting one number from the other does not tell you how many bots visited. Start by checking what each number counts.

1. Prepare a safe sample

Prompt

Help me prepare seven days of aggregate traffic for my own site. Suggest columns for time bucket, page category, request count, analytics sessions and host bot labels if available. Exclude IPs, IDs, cookies, query strings and full user-agent strings. Label both date ranges and the timezone.

2. Keep the categories honest

Prompt

Using [aggregate rows], report labeled crawler requests, other labeled requests and unclassified requests. Keep analytics sessions separate. Explain the label method only if supplied. Do not treat all unflagged requests as people.

3. Decide what to measure next

Prompt

From [weekly summary], name two observations and one alternative explanation for each. Suggest a reader action I can track next week. Flag supported differences in consent, caching, instrumentation or time windows. Do not change access settings.

Try it on one example

If labeled crawler requests rise while analytics sessions stay flat, report both. You still do not know whether a crawler led any readers to the site.

Inspired by Crawler Zoo: https://www.reddit.com/r/InternetIsBeautiful/s/L92dqs9H4c.

Use it for: Compare aggregate host and analytics data without guessing how much traffic is human.

Back to the archive