> ## Content Index > Fetch the complete content index at: https://blog.sudowl.com/llms.txt > Use this file to discover other available public pages before exploring further. # Bots vs. Real Readers: Counting Blog Visitors From the Access Log - URL: https://blog.sudowl.com/bots-vs-real-readers-access-log/ - Published: 2026-10-22T15:00:00.000Z - Updated: 2026-10-25T17:42:08.000Z - Description: My blog logged 140 to 500 hits a day. Real readers: 15 to 61. The four filter rules I use on the access log, plus the small Python script and cron job. - Author: Lukas - Tags: Linux, nginx My blog has no analytics script. No Google Analytics, no Plausible, no cookie banner. The about page says "no tracking" and I want that to stay true. But I still wanted to know if anybody actually reads this stuff. The web server writes every request into the access log anyway. So I count from there. First look at the raw numbers was nice for the ego. Somewhere between 140 and 500 hits on posts per day. Then I filtered out the bots, and in the first week it was 15 to 61 real readers a day. That's the real number, and that's what this post is about. **TL;DR:** Don't count raw hits, most of them are bots. Count a hit as a reader only if the user agent isn't a bot, the same visitor (IP + user agent) also loaded CSS/JS/images that day, they didn't open more than 20 posts that day, and the IP never touched `/ghost/` (that's me in the admin). A small Python script does this once a day from cron and writes `date,slug,views,unique` into a CSV. No cookies, no JS, no IPs stored. ## The four rules Here's what actually filters the junk out, in the order it matters: 1. **Bot user agents.** Anything with `bot`, `crawl`, `spider`, `curl`, `python`, `headless`, `feed` and so on in the user agent is out. That catches the honest ones. 2. **Must load assets.** A real browser that opens a post also fetches the theme CSS, JS and images. Most crawlers only grab the HTML. So a visitor (IP + user agent) only counts if they also requested something under `/assets/`, `/public/` or `/content/images/` that same day. 3. **More than 20 posts a day = scraper.** Some scrapers send a normal Chrome user agent and even load assets. But no human reads 30 of my posts in one day. Above 20 distinct posts per visitor per day, the whole visitor is dropped for that day. 4. **Exclude me.** Any IP that opened `/ghost/` (the Ghost admin) is me, or someone I'd rather not count. Those IPs are left out. Rule 2 is the big one. Rule 1 alone still leaves a lot of fake traffic, because plenty of bots pretend to be a browser. ## The log line Apache sits in front of the Ghost container as a reverse proxy (setup is in [Apache as a reverse proxy for Docker apps](/apache-reverse-proxy-docker-apps/)). The log is the normal combined format: ```text 203.0.113.10 - - [20/Oct/2026:14:02:11 +0000] "GET /some-post/ HTTP/2.0" 200 18234 "https://www.example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ..." ``` IP, date, request, status, user agent. That's all I need. nginx writes the same format by default, so this works there too. ## The script This is a cut-down version of what runs on the server. The real one also reads rotated `.gz` logs and two log folders, because the blog moved to blog.sudowl.com and the old domain's logs still hold the history. ```python #!/usr/bin/env python3 import csv, re, sys LINE = re.compile(r'^(\S+) \S+ \S+ \[(\d{2})/(\w{3})/(\d{4}):[^\]]+\] ' r'"GET (\S+) HTTP/[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"') MONTHS = {m: i for i, m in enumerate('Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec'.split(), 1)} BOT = re.compile(r'bot|crawl|spider|slurp|curl|wget|python|go-http|java/|okhttp|headless|' r'lighthouse|preview|facebookexternalhit|feed|rss|monitor|uptime|scan|' r'axios|libwww|ahrefs|semrush|bing|yandex|petal', re.I) ASSET = re.compile(r'^/(assets|public|content/images)/') SKIP = re.compile(r'^/(ghost|content|assets|public|members|rss|tag|author|page|sitemap|favicon|robots)(/|$|\.|-)') SLUG = re.compile(r'^/([a-z0-9][a-z0-9-]*)/$') MAX_POSTS_PER_DAY = 20 hits, assets, admins = [], set(), set() for line in open(sys.argv[1], errors='replace'): m = LINE.match(line) if not m: continue ip, dd, mon, yyyy, url, status, ua = m.groups() day = f'{yyyy}-{MONTHS.get(mon, 0):02d}-{dd}' path = url.split('?', 1)[0] if path.startswith('/ghost/'): admins.add(ip) continue if ASSET.match(path): assets.add((day, ip + '|' + ua)) continue if status not in ('200', '304') or BOT.search(ua) or SKIP.match(path): continue s = SLUG.match(path) if s: hits.append((day, ip, ua, s.group(1))) # how many different posts did each visitor open per day posts_per_visitor = {} for day, ip, ua, slug in hits: posts_per_visitor.setdefault((day, ip + '|' + ua), set()).add(slug) agg = {} for day, ip, ua, slug in hits: v = ip + '|' + ua if (day, v) not in assets or ip in admins or len(posts_per_visitor[(day, v)]) > MAX_POSTS_PER_DAY: continue rec = agg.setdefault((day, slug), [0, set()]) rec[0] += 1 rec[1].add(v) w = csv.writer(sys.stdout) w.writerow(['date', 'slug', 'views', 'unique']) for (day, slug), (views, visitors) in sorted(agg.items()): w.writerow([day, slug, views, len(visitors)]) ``` A few things worth pointing out: - The `/ghost/` check runs before anything else, so an admin IP lands on the list no matter what else it requested. - `SKIP` throws out tag pages, author pages, the sitemap and so on. I only want posts. `SLUG` then matches `//`, which is how Ghost post URLs look. - `views` counts reloads, `unique` counts distinct IP + user agent per post and day. Somebody who opens a post three times is 3 views, 1 unique. - The IP+UA pairs only live in memory while the script runs. The CSV has date, slug and two numbers. No IPs. ## The cron job The real script recomputes "yesterday in UTC" by default and replaces that day's rows in the CSV. It writes to a temp file and renames it, so a crash in the middle doesn't leave half a CSV. Running it twice gives the same result, which is how cron jobs should be ([more on that here](/idempotency-cron-automation-duplicate-uploads/)). ```text # /etc/cron.d/blog-views 15 0 * * * root /usr/bin/python3 /opt/blog-stats/blog-views.py ``` 00:15 UTC, so the day is over and the log has everything. There's also a `--all` mode that rebuilds every day still in the logs. I needed that once after the domain change, when the script was still only reading the old domain's log and the numbers suddenly stopped. ## Gotcha: the logs don't keep forever My access logs rotate weekly and keep 4 weeks. After that they're gone. So the CSV is the history, not the logs. If you only ever run "rebuild from logs", you lose everything older than your rotation window. Check your `logrotate` settings before you rely on old logs ([logrotate on Linux](/logrotate-linux-log-rotation/)). ## What came out Readers per day, from the CSV, first 11 days: | Day | Readers | | ----- | ------- | | 10-11 | 61 | | 10-12 | 27 | | 10-13 | 32 | | 10-14 | 15 | | 10-15 | 19 | | 10-16 | 37 | | 10-17 | 28 | | 10-18 | 27 | | 10-19 | 40 | | 10-20 | 104 | | 10-21 | 37 | That's counted per post and day. If one person reads three posts, that's 3 in this table. So the number of actual humans is a bit lower still. Yes, these numbers are small. That's fine. That's the point. Before I filtered, the log said hundreds a day, and I would have believed it. Now I know a normal day is 15 to 40 people, and a really good day gets past 100. The most read post in that time was [Tailscale for a friends-only game server](/tailscale-gameserver-friends-only/), 20 views from 19 readers. Third place wasn't a server post at all, it was the [HP Victus GPU stuck at 30 W](/hp-victus-gpu-stuck-30w-hw-power-brake-ec-reset/) one. A laptop problem I fixed for myself and wrote down. People with a real problem search for it, find the post, read it. That tells me more about what to write next than the raw hit count ever did. ## Gotcha: shared IPs and changing IPs The visitor key is IP + user agent. That's not perfect: - Two people behind the same office or carrier NAT with the same browser version count as one. - One person on mobile who switches from Wi-Fi to mobile data counts as two. - If I read my blog from a network where I never opened `/ghost/`, I count myself. None of this matters much at this size. It's a rough number, not an exact one. But it's honest, and it's rough in both directions instead of always inflating like raw hits do. ## Gotcha: /ghost/ is not only the admin Ghost serves its public Content API under `/ghost/api/content/` too, and the built-in search uses it. So a reader who opens the search box could end up on my admin list and get filtered out. With my numbers that's maybe a reader here and there. If you copy this, it's cleaner to only treat `/ghost/api/admin/` or the admin page itself as "that's me". ## Why not just use analytics JS - **Privacy.** No cookies, no third-party script, nothing in the browser. The log exists anyway, and I only keep aggregates. - **Ad blockers.** This blog is for tech people, and a lot of them block analytics scripts. The access log doesn't care. - **Bots don't run JS.** True, and that's the one thing JS analytics does better out of the box. The asset rule gets me most of the way there. What I don't get: time on page, scroll depth, where people click. I don't need it. I want to know which posts get read and if the number goes up. For server health and uptime I use proper monitoring anyway ([Grafana vs Netdata vs Uptime Kuma](/grafana-vs-netdata-vs-uptime-kuma/)), that's a different job. The blog itself had a rough start, by the way. The whole thing got rebuilt after the database was gone ([rebuilding my Ghost blog](/ghost-blog-rebuild-after-database-loss/)). Seeing 104 real readers on one day after that felt better than any 500 hits from bots. --- ### Related posts - [Moving a Blog to a New Domain Without Losing Google: 301s, Canonicals, Search Console](https://blog.sudowl.com/move-blog-new-domain-301-search-console/) - [Let's Encrypt Kept Failing After a Server Move. The Culprit Was a Second A Record.](https://blog.sudowl.com/lets-encrypt-fails-duplicate-a-record/) - [Caddy vs Nginx — When 5 Lines Beats certbot + Cron](https://blog.sudowl.com/caddy-vs-nginx-auto-https/)