> ## Content Index > Fetch the complete content index at: https://blog.sudowl.com/llms.txt > Use this file to discover other available public pages before exploring further. # The Idempotency Bug That Uploaded the Same File 153 Times - URL: https://blog.sudowl.com/idempotency-cron-automation-duplicate-uploads/ - Published: 2026-07-15T10:00:00.000Z - Updated: 2026-10-11T10:16:27.000Z - Description: A cron job re-ran a side effect every two minutes because its "done" marker was gated on a flaky downstream step. The fix is one line moved up. Here's the pattern. - Author: Lukas - Tags: Linux, Docker One of my automations quietly dumped **over 1,500 duplicate files** into cloud storage from a *single* incoming email. Same ten attachments, uploaded 153 times, spread across folders, over about five hours. Nothing crashed. No error was raised that anyone acted on. It just kept politely doing the wrong thing every two minutes until I noticed. The root cause is one of those bugs that's obvious in hindsight and invisible in review, and it comes down to a single question you should ask of every cron job you own: **if this runs twice, what happens?** **TL;DR:** The job did an irreversible side effect (upload) *before* it marked the work as done, and it only marked the work as done *if a fragile later step succeeded*. The later step kept failing, so "done" was never written, so the next run repeated the upload. Fix: commit the done-marker right after the side effect, not after the fragile step. ## The setup Simplified, the job looked like this. A cron fires every two minutes, finds unprocessed items, does something with side effects, then triggers a heavier downstream step: ```python def run(): processed = load_state() # ids we've already handled for item in find_unprocessed(): if item.id in processed: continue upload_attachments(item) # <-- side effect: writes to cloud ok = trigger_downstream(item, timeout=180) # <-- flaky: 180s call, often times out if ok: # <-- only here do we record "done" mark_read(item) processed.add(item.id) save_state(processed) ``` Read that `if ok:` again. The entire dedup mechanism — `mark_read`, `processed.add`, `save_state` — is gated on `trigger_downstream` returning success. ## What actually happened `trigger_downstream` was a 180-second call to a much heavier process. Under load it kept timing out and returning `ok = False`. So: 1. Run at 10:00 — upload the attachments, trigger downstream, it times out, **nothing marked done**. 2. Run at 10:02 — the item is *still* unprocessed (we never marked it), so… upload the attachments **again**, trigger, time out again. 3. Run at 10:04 — same thing. And again. And again. 153 runs × \~10 attachments ≈ 1,530 duplicate files. The upload was cheap and reliable; the thing I gated on was expensive and flaky. I had it exactly backwards. The tell in the logs was brutal once I looked: **151 timeouts, 2 successes** for that one item, and an upload-count that climbed in a perfect straight line every two minutes. ## The fix is one idea, moved up The irreversible side effect must be committed to state **as soon as it happens**, independent of anything downstream. Rewrite: ```python def run(): processed = load_state() for item in find_unprocessed(): if item.id in processed: continue upload_attachments(item) # commit "done" immediately — the upload happened, it must never repeat processed.add(item.id) save_state(processed) mark_read(item) ok = trigger_downstream(item, timeout=180) if not ok: log.warning("downstream failed for %s — will NOT re-upload; needs manual retry", item.id) ``` Now a downstream failure costs you *one dropped downstream action* that you can see in the logs and retry — not an infinite loop of duplicated side effects. That's the trade you want: **a flaky step should fail forward, not backward into the part that already touched the world.** ## The general rule > Order your steps so the irreversible one is protected by the durable marker, and never gate that marker on a step that can fail for boring reasons (timeouts, rate limits, a service being briefly down). A cron job is just a function the OS calls again and again. It *will* be interrupted mid-run, retried, and overlapped. If re-running it repeats a side effect, that's not an edge case, that's Tuesday. ## Belt and suspenders Moving the marker up fixes this specific bug. If you want the side effect itself to be safe under races, make it idempotent so a duplicate run is a no-op instead of a second copy: - **Deterministic keys.** Upload to a path derived from the item, e.g. `sender//`, not a fresh timestamped name each run. A repeat overwrites instead of multiplying. (My bug was made *worse* by timestamped, always-unique filenames — every run was guaranteed to look "new".) - **Check-before-write** when the target supports it: does an object with this key already exist? Skip. - **A lock** so two runs can't overlap. On Linux, wrap the job in `flock` and a slow run simply skips the next tick instead of stacking: ```bash * * * * * /usr/bin/flock -n /tmp/myjob.lock /usr/local/bin/myjob.sh ``` - **A dead man's switch.** The reason this ran for five hours is that "success" was the only thing that pinged me. A job that also alerts on *repetition* — "the same item processed N times" — would have caught it in minutes. Alert on the anomaly, not just the failure. None of these are exotic. They're the difference between a cron job that heals and one that amplifies. ## FAQ **Isn't the real bug the flaky downstream call?** That's *a* bug, but it's the wrong one to fix first. Downstream calls will always fail sometimes — timeouts, restarts, rate limits are normal. The design flaw is letting a normal failure feed back into a side effect. Fix the ordering; then, separately, make the downstream call more robust. **Why not just mark the item read at the very start?** You can, and for pure side effects it's often right. The subtle point is *which* action you're protecting. Commit the marker immediately after the step that touches the outside world irreversibly, so no retry can repeat that step. Anything after it is safe to lose and retry. **How do I retro-fit this without re-processing everything?** Add the durable marker first (so the bleeding stops), then reconcile once: list what's actually in the target, dedupe by your deterministic key, keep one. I trashed \~1,520 duplicates keeping one of each — reversible via the trash, in case a "duplicate" wasn't. **Does this only apply to cron?** No. Anything retried — message queues with at-least-once delivery, webhook receivers, CI steps, a user double-clicking submit. "At-least-once delivery" is the norm, so "safe to run more than once" is a requirement, not a nicety. If you're wiring up scheduled jobs on Linux, [systemd timers instead of cron](/systemd-timers-instead-of-cron/) gives you per-run logging and `RemainAfterExit` semantics that make this kind of state bug easier to spot in the first place. --- ### Related posts - [Grafana vs Netdata vs Uptime Kuma — I Run All Three, Here's What Each Is Actually For](https://blog.diengdoh.com/grafana-vs-netdata-vs-uptime-kuma/?ref=blog.sudowl.com) - [Docker vs Podman — What's the Difference and Should You Switch?](https://blog.diengdoh.com/docker-vs-podman/?ref=blog.sudowl.com) - [Coolify vs Dokploy vs CapRover — Self-Hosted PaaS Compared (2026)](https://blog.diengdoh.com/coolify-vs-dokploy-vs-caprover/?ref=blog.sudowl.com)