> ## Content Index > Fetch the complete content index at: https://blog.sudowl.com/llms.txt > Use this file to discover other available public pages before exploring further. # Proxmox + ZFS — A Storage Pool That Survives a Disk Death - URL: https://blog.sudowl.com/proxmox-zfs-storage-pool-setup/ - Published: 2026-06-07T10:00:00.000Z - Updated: 2026-10-11T10:16:27.000Z - Description: Build a redundant ZFS pool on Proxmox VE — mirror vs raidz, ashift, ARC cap, scrubs, and how to replace a dead disk without losing a VM. - Author: Lukas - Tags: Linux A disk will die. The only question is whether your VMs notice. On Proxmox I run ZFS for exactly one reason: I can yank a failing drive, slot a new one, and resilver while everything stays online. **TL;DR:** ```bash # create a mirror from raw disks (by-id, ALWAYS) zpool create -o ashift=12 tank mirror \ /dev/disk/by-id/ata-WDC_WD40EFZX_AAA \ /dev/disk/by-id/ata-WDC_WD40EFZX_BBB zfs set compression=lz4 atime=off tank pvesm add zfspool tank-vms -pool tank -content images,rootdir -sparse 1 ``` Details below. ## Use by-id, not /dev/sdX `/dev/sda` is not stable. Reboot, add a USB stick, swap a cable — the kernel reassigns letters and now your pool references the wrong disk. ZFS handles this at runtime, but you do not want to be reading `zpool status` at 2am trying to figure out which physical drive `sdc` is today. ```bash ls -l /dev/disk/by-id/ | grep -v part ``` Use the `ata-*` or `wwn-*` paths. They're printed on the drive label and survive everything. If you already built a pool with `/dev/sdX`, fix it: ```bash zpool export tank zpool import -d /dev/disk/by-id tank ``` `ashift=12` forces 4K sectors. Every modern drive is 4Kn or 512e-pretending-to-be-512\. Get this wrong at create time and you cannot change it later — you'd have to destroy and rebuild the pool. Always `ashift=12`. ## mirror vs raidz1 vs raidz2 — pick the right one This is the decision that matters. You cannot change a vdev's layout after creation. - **mirror** (2 disks): one can die. Best IOPS, simplest to grow (add more mirror vdevs), fastest resilver. Lose 50% capacity. My default for VM storage. - **raidz1** (3+ disks): one can die. \~67%+ usable. Slower, and resilver on big disks is scary — a second drive often fails *during* the multi-hour rebuild. Fine for 4TB-class disks, sketchy above \~8TB. - **raidz2** (4+ disks): two can die. The safe choice for large arrays. Survives a second failure mid-resilver. ```bash # 4-disk raidz2 for a bulk/backup pool zpool create -o ashift=12 tank raidz2 \ /dev/disk/by-id/ata-DISK1 /dev/disk/by-id/ata-DISK2 \ /dev/disk/by-id/ata-DISK3 /dev/disk/by-id/ata-DISK4 ``` For VM workloads I run mirrors. For a backup target where IOPS don't matter, raidz2\. Never raidz1 on disks bigger than 8TB. ## Dataset properties Set these on the pool root so they inherit down: ```bash zfs set compression=lz4 tank # free; almost always a net win zfs set atime=off tank # stop writing on every read ``` `lz4` compression is nearly free on CPU and skips incompressible data automatically. Turn it on everywhere. **recordsize** matters for VM zvols. The default `128k` is right for general files but wrong for a VM doing 4K-16K random IO inside its guest filesystem — you get read/write amplification. For VM-heavy pools, set the zvol `volblocksize` to `16k`–`64k`: ```bash # Proxmox controls this per-storage; set the default block size pvesm set tank-vms -blocksize 32k ``` For a dataset holding large media, push the other way with `recordsize=1M`. ## Add it to Proxmox GUI: **Datacenter > Storage > Add > ZFS**, pick `tank`, content = Disk image + Container, tick **Thin provision**. Or CLI: ```bash pvesm add zfspool tank-vms -pool tank -content images,rootdir -sparse 1 ``` `-sparse 1` is thin provisioning — a 100G disk only consumes what's actually written. Combined with snapshots this is the whole reason to bother. See my [Ubuntu cloud-init template post](https://blog.sudowl.com/proxmox-cloud-init-vm-template-ubuntu/) for spinning up VMs that land straight on this pool. ## Cap the ARC before it eats your RAM ZFS uses a memory cache (ARC) that by default grabs up to **half your RAM**. On a Proxmox box that RAM is for VMs, and the ARC counts as "used" — you'll see memory pressure and wonder why. Cap it. Rule of thumb: give the ARC what you can spare, often 4–16 GB on a homelab node. Here, 8 GB: ```bash # /etc/modprobe.d/zfs.conf options zfs zfs_arc_max=8589934592 ``` ```bash echo 'options zfs zfs_arc_max=8589934592' > /etc/modprobe.d/zfs.conf update-initramfs -u reboot ``` That number is bytes (8 \* 1024^3). Check it took: `cat /sys/module/zfs/parameters/zfs_arc_max`. ## THE big gotcha: never put ZFS on hardware RAID ZFS wants **raw disks**. It does its own checksumming, redundancy, and error recovery, and it assumes it can see and trust the physical drives directly. Put it on top of a hardware RAID controller and you break all of that: the controller hides drives behind its own RAID, lies about write completion via its cache, and remaps sectors silently. When ZFS detects corruption it can no longer find a good copy to heal from, because the layer underneath it scrambled the topology. You get the overhead of ZFS with none of the safety. If your server has a RAID card, flash it to **IT mode** (HBA / passthrough) so it presents disks raw. LSI 9211-8i and friends do this fine. No HBA, no IT mode — use the onboard SATA ports directly. Do not configure a single-disk RAID0 per drive as a workaround; the cache and remapping problems remain. **The exception:** a true HBA in IT mode is exactly what you want — it's a dumb pass-through, which is the point. ## ECC RAM — nice, not mandatory ZFS doesn't *require* ECC RAM, despite what forum threads claim. ECC protects in-flight data in memory from bit flips, which is good hygiene for any storage server. But non-ECC ZFS is still far safer than non-ECC anything-else, because ZFS at least checksums data on disk. If your hardware has ECC, great. If it doesn't, run ZFS anyway — just don't lose sleep over it. ## Scrub monthly A scrub reads every block and verifies checksums, healing silent corruption from the redundant copy. On Debian/Proxmox the package ships a systemd timer: ```bash systemctl enable --now zfs-scrub-monthly@tank.timer systemctl list-timers | grep zfs-scrub ``` Prefer cron? One line: ```cron 0 3 1 * * /sbin/zpool scrub tank ``` Read the result with `zpool status`. `state: ONLINE` and `errors: No known data errors` is what you want. Any non-zero in the READ/WRITE/CKSUM columns means a disk is going. ## Replace a dead disk The whole point. Say `zpool status` shows a drive `DEGRADED` or `FAULTED`: ```bash zpool offline tank /dev/disk/by-id/ata-WDC_WD40EFZX_BBB ``` Physically swap the drive. Find the new disk's by-id path, then: ```bash zpool replace tank \ ata-WDC_WD40EFZX_BBB \ /dev/disk/by-id/ata-WDC_WD40EFZX_CCC ``` ZFS resilvers — rebuilds the new disk from the surviving copy. Watch it: ```bash zpool status -v tank ``` You'll see `resilver in progress` with a percentage and ETA. VMs keep running the whole time. When it hits `errors: No known data errors` and the pool is `ONLINE`, you're done. On a mirror this is fast; on raidz with large disks it can take a day — which is exactly why raidz1 + huge disks is a bad bet. ## Snapshots are not backups Snapshots are instant and nearly free, and they live on the *same pool*: ```bash zfs snapshot tank/vm-101-disk-0@before-upgrade zfs list -t snapshot zfs rollback tank/vm-101-disk-0@before-upgrade ``` Take one before every risky change. But a snapshot dies with the pool — if the chassis burns or the controller cooks every drive, your snapshots burn with it. They are not a backup. For actual backups, push VMs to a separate box with [Proxmox Backup Server](https://blog.sudowl.com/proxmox-backup-server-setup/), and ship an offsite copy with [restic to S3/MinIO](https://blog.sudowl.com/restic-backup-s3-minio-encrypted/). One pool of redundant disks plus snapshots covers a disk death; it does not cover a host death. ## Verify ```bash zpool status -v # ONLINE, no errors, all disks present zfs list # datasets + space, thin usage zpool list # pool capacity and health zfs get compression,atime,recordsize tank ``` If `zpool status` is `ONLINE` with zero errors and your properties stuck, the pool is doing its job — and the next dead disk is a swap, not an outage. --- ### Related posts - [Lower Your DNS TTL Before You Migrate a Server (A Checklist)](https://blog.diengdoh.com/lower-dns-ttl-before-migration/?ref=blog.sudowl.com) - [Reskinning Ghost's Source Theme with Nothing but Code Injection](https://blog.diengdoh.com/ghost-source-theme-code-injection-css/?ref=blog.sudowl.com) - [Serving a Static SPA on Apache: .htaccess Fallback, Caching, and the Container Nobody Used](https://blog.diengdoh.com/static-spa-apache-htaccess-fallback/?ref=blog.sudowl.com)