> ## Content Index > Fetch the complete content index at: https://blog.sudowl.com/llms.txt > Use this file to discover other available public pages before exploring further. # Encrypting Game Server Backups Before They Leave the Node - URL: https://blog.sudowl.com/encrypt-game-server-backups-panel-managed-key/ - Published: 2026-10-26T10:00:00.000Z - Updated: 2026-11-05T18:02:37.000Z - Description: Backup encryption in my open-source Hytale panel: ChaCha20-Poly1305 streaming, key never on the node, key rotation, old backups still restorable. - Author: Lukas - Tags: Security, Hytale, Gameserver I build TalePanel, an open-source server panel for Hytale. It can push backups off-site: S3, SFTP, WebDAV, cloud drives. Until 26.10\. those backups were plain zip files. That means anyone who gets the S3 key or the SFTP login for the backup target gets every world on the server. Player data, builds, configs, all of it. That bugged me for a while. Off-site backups are the right idea, but you're copying your data to a place you control less than your own server. So on 26.10\. I sat down and made backups encrypted by default. This post is how it works and what I learned on the way. **TL;DR:** Encrypt on the node, at the moment the archive is created, so the local copy and every upload are already ciphertext. Use a streaming AEAD (I used ChaCha20-Poly1305 STREAM, 64 KiB chunks) so big worlds don't need to fit in RAM and tampering or truncation fails loudly. Keep the key on the panel, never on the node. Store a key id in every archive and in the database so you can rotate keys and still restore old backups. Detect old plaintext backups by the missing header so nothing breaks. ## Why not just encrypt the upload The easy version is: rclone has a crypt remote, put it in front of the upload, done. I didn't want that for two reasons. First, the local copy on the node stays plaintext. If someone gets into the node, they don't even need the backup credentials. Second, every backup target would need its own crypt setup. Users configure targets in the panel, and I don't want them to think about encryption at all. It should just be on. So the rule became: the archive is encrypted when it's created. Everything after that, local disk, rclone, the cloud, only ever sees ciphertext. The rclone transfers didn't change at all. ## The format Each encrypted backup starts with a small fixed header, then the encrypted data: ```text TPBKENC1 8 bytes magic, marks an encrypted TalePanel backup key id 1 byte which key sealed this archive nonce prefix 7 bytes random per archive chunks ... ChaCha20-Poly1305 STREAM, 64 KiB plaintext + 16-byte tag each ``` Why STREAM and not one big AEAD call: a Hytale world can be gigabytes. One-shot AEAD means the whole thing in memory, or no authentication until the very end. STREAM splits it into chunks, each with its own tag, and the last chunk is marked as last. So: - a flipped byte anywhere fails on that chunk - a cut-off file fails because the "last" chunk never comes - memory use stays at two 64 KiB buffers The nonce is the 7-byte random prefix plus a counter and a last-block flag that STREAM manages itself. Random prefix per archive means two backups under the same key never share nonces. The daemon is written in Rust, so this is the RustCrypto `chacha20poly1305` crate with its stream helpers. In Go or Python you'd find the same building blocks. ## Gotcha: zip wants to seek The daemon builds the zip and then has to encrypt it. My first idea was to pipe the zip writer straight into the encryptor. Doesn't work: the zip writer needs `Seek`, because it goes back to patch headers after writing file data. An encrypting stream can't seek. So it's zip to a temp file first, then encrypt the temp file into the final archive, then delete the temp file. A bit more disk I/O, but simple and correct. If your panel does the same, keep the temp file on the same disk and make sure it gets deleted on errors too. ## Where the key lives This was the part I thought about longest. The node needs the key to encrypt and decrypt. But if the key sits on the node, a compromised node gives you key and backups together, and we're back at square one. What I did: - The panel owns a keyring: one active key plus older keys kept for rotation. It's created on first use. - The keyring is stored encrypted in the panel database (AES-256-GCM under a separate key from the panel's environment, the same one that protects TOTP secrets). - For every create, restore or download, the panel sends the right key to the daemon in the request body. The daemon uses it and forgets it. It's never written to disk on the node. - Each backup row in the database records `enc_key_id`. The archive header has the same id. Restore then looks up the backup's key id, sends that key, and the daemon decrypts. Download works the same way through a POST. A plain GET download of an encrypted archive is refused, because without the key it would just hand you ciphertext. ## Rotation without losing old backups Admins (owner role only) get three endpoints in the panel: show key status, export the key, rotate. All three are audit-logged. Rotate makes a new active key. Old keys stay in the keyring. New backups use the new key, old backups still have their old key id, so they still restore. That's the whole point of the one-byte key id in the header. Export exists for one reason: escrow. If the panel database is gone, the backups are useless without the key. So the UI tells owners to export the key and store it somewhere off the panel. A password manager is fine. Encrypted backups with the only key on the same box that died are a nice way to lose everything twice. ## Gotcha: old backups Every existing install already had plaintext zip backups. Those must not break. The fix is cheap: read the first 8 bytes. `TPBKENC1` means encrypted, anything else means legacy zip and it passes through unchanged. Restore and download work for both. No migration of old archives, no flag day. One exception I made on purpose: when you move a server from one node to another, the panel builds a temporary transfer archive. That one stays plaintext, because it only exists for the move and is deleted afterwards. Encrypting it would add a key round-trip for no real gain. ## Tests that mattered I wanted tests for the ugly cases, not just "encrypt then decrypt works": - round trip for an empty file, a small file, several chunks, and a file that's an exact multiple of 64 KiB (the classic off-by-one around the last chunk) - one flipped byte must fail - wrong key must fail - truncated file must fail - a legacy plaintext zip must pass through - create an encrypted backup, then restore it, end to end Then a real run against a WebDAV target: backup, upload, download, restore. The restored files were byte-identical to the originals. That's the test I trust most. ## What's still open on the transport The key travels from panel to daemon in a request. That link needs to be properly secured, otherwise you've just moved the problem. On 26.10\. I looked at that path and the panel to daemon link was still plain HTTP inside the node network, even though our SECURITY.md already said HTTPS. Docs and code didn't match. I fixed the docs that day and agreed on a design for mutual TLS: the panel acts as a small private CA, every daemon gets a client cert, there's a transition period so existing nodes keep working, and the existing bearer token stays as a second factor. That shipped a few days later in 0.12.0-beta. Lesson for me: write the security doc from the code, not from the plan. ## Should you do this for your own server If you run a single Hytale or Minecraft server and back up with a script, you don't need a panel feature. Use restic, it encrypts by default and handles all of this for you, see [encrypted restic backups to S3](/restic-backup-s3-minio-encrypted/). For the basic backup and restart setup, see [Hytale backups with systemd](/hytale-server-backups-auto-restart-systemd/). If you build or run a panel, the short version is: 1. Encrypt at creation, not at upload. 2. Streaming AEAD with per-chunk tags. 3. Key on the panel, never at rest on the node. 4. Key id in every archive so rotation is painless. 5. Detect old plaintext and keep it working. 6. Tell people to export the key. And whatever you do, check that the backups actually run. Encrypted backups that silently stopped three weeks ago don't help anyone, see [backups that are silently dead](/backups-silently-dead-check-backup-freshness/). If you're curious how I run the panel itself, there's [a Hytale server with Docker and a panel](/hytale-server-docker-panel/). TalePanel is AGPL and the code is on GitHub (BitaceS/talepanel) if you want to look at the crypto part yourself. --- ### Related posts - [Bridging a Hytale Forum and Discord Without Making a Mess](https://blog.sudowl.com/hytale-forum-discord-bridge/) - [Hytale Server Files Explained: Where Your World Actually Lives](https://blog.sudowl.com/hytale-server-files-explained/) - [How Much RAM Does a Hytale Server Need? The Formula I Actually Use](https://blog.sudowl.com/hytale-server-ram-how-much/)