> 3. Park a small reasonably low-power computer at a friend's house across town or somewhere a little further out -- it can be single-disk or raidz1. Send ZFS snapshots to it using Tailscale or whatever. (And scrub that regularly, too.)

Maybe I’m hanging out in the wrong circles, but I would never think it appropriate to make such a proposal to a friend; “hey let me set up a computer in your network, it will run 24/7 on your power and internet and I’ll expect you to make sure it’s always online, also it provides zero value to you. In exchange I’ll give you some unspecified amount of pizza, like a pointy haired boss motivating some new interns”.

> In exchange I’ll give you some unspecified amount of pizza

You mean, in exchange we will have genuine social interactions that you will value much more highly than the electricity bill or the pizza.

Plus you will be able to tease me about my overengineered homelab for the next decade or more.

About the worst I can imagine happening (other than the new-found ability to rockroll someone's TV as a prank) is that said friend might take an interest in how I manage my data and want a hand with setting up a similar thing for themselves.

And that's all fine too. I like my friends quite a lot, and we often help eachother do stuff that is useful: Lending tools or an ear to vent at, helping to fix cars and houses, teaching new things or learning them together, helping with backups -- whatever. We've all got our own needs and abilities. It's all good.

Except... oh man: The electric bill! I forgot about that.

A small computer like what I'm thinking would consume an average of less than 10 Watts without optimization. That's up to nearly $16 per year at the average price of power in the US! I should be more cognizant of the favors I request, lest they cause my friends to go bankrupt.

/s, of course, but power can be a concern if "small" is misinterpreted.

Or find someone else with a similar backup need and then both just agree to have enough space to host remote backups for the other. I would have to increase my ZFS from N to 2N TB, but that would be less work and cheaper than setting up a backup computer for N TB somewhere else.

I have a simpler approach that I've used at home for about 2 decades now pretty much unchanged.

I have two raid1 pairs - "the old one", and "the new one", plus a third drive the same sizes as "the old pair". The new pair is always larger than the old pair, in the early days it was usually well over twice as big but drive growth rates have slowed since then. About every three years I buy a new "new pair" + third drive, and downgrade the current "new pair" to be the4 "old pair". The old pair is my primary storage, and gets rsynced to a partition that's the same size on the new pair. Te remainder of the new pair is used for data I'm OK with not being backed up (umm, all my BitTorrented Linux isos...) The third drive is on a switched powerpoint and spins up late Sunday night and rsyncs the data copy on the new pair then powers back down for the week.

>3. Park a small reasonably low-power computer at a friend's house across town or somewhere a little further out -- it can be single-disk or raidz1. Send ZFS snapshots to it using Tailscale or whatever. (And scrub that regularly, too.)

Unless you're storing terabyte levels of data, surely it's more straightforward and more reliable to store on backblaze or aws glacier? The only advantage of the DIY solution is if you value your time at zero and/or want to "homelab".

A chief advantage of storing backup data across town is that a person can just head over and get it (or ideally, a copy of it) in the unlikely event that it becomes necessary to recover from a local disaster that wasn't handled by raidz and local snapshots.

The time required to set this stuff up is...not very big.

Things like ZFS and Tailscale may sound daunting, but they're very light processes on even the most garbage-tier levels of vaguely-modern PC hardware and are simple to get working.

yep, also I'm scared of AWS holding my data hostage if I ever decide to travel to Iran

I'd much rather just have a backblaze solution and maybe redundant local backups with Time Machine or your local backup of choice (which work fine for terabytes at this point). Maybe create a clone data drive and drop it off with a friend every now and then which should capture most important archive stuff.

If you mostly care about data integrity, then a plain RAID-1 mirror over three disks is better than RAIDZ. Correlated drive failures are not uncommon, especially if they are from the same batch.

I also would recommend an offline backup, like a USB-connected drive you mostly leave disconnected. If your system is compromised they could encrypt everything and also can probably reach the backup and encrypt that.

Better how?

With RAID 1 (across 3 disks), any two drives can fail without loss of data or availability. That's pretty cool.

With RAIDZ2 (whether across 3 disks or more than 3; it's flexible that way), any two drives can fail without loss of data or availability. At least superficially, that's ~equally cool.

That said: If something more like plain-Jane RAID 1 mirroring is desired, then ZFS can do that instead of RAIDZ (that's what the mirror command is for).

And it can do this while still providing efficient snapshots (accidentally deleted or otherwise ruined a file last week? no problem!), fast transparent data compression, efficient and correct incremental backups, and the whole rest of the gamut of stuff that ZFS just boringly (read: reliably, hands-off) does as built-in functions.

It's pretty good stuff.

All that good stuff works fine with single disks, too. Including redundancy: ZFS can use copies=2 to store multiple (in this case, 2) copies of everything, which can allow for reading good data from single disks that are currently exhibiting bitrot.

This property carriers with the dataset -- not the pool. In this way, a person can have their extra-important data [their personal writings, or system configs from /etc, or whatever probably relatively-small data] stored with extra copies, and their less-important (probably larger) stuff stored with just one copy...all on one single disk, and without thinking about any lasting decisions like allocating partitions in advance (because ZFS simply doesn't operate using concepts like hard-defined partitions).

I agree that keeping an offline backup is also good because it provides options for some other kinds of disasters -- in particular, deliberate and malicious disasters. I'd like to add that this this single normally-offline disk may as well be using ZFS, if for no other reason than bitrot detection.

It's great to have an offline backup even if it is just a manually-connected USB drive that sits on a shelf.

But we enter a new echelon of bad if that backup is trusted and presumed to be good even when it has suffered unreported bitrot:

Suppose a bad actor trashes a filesystem. A user stews about this for a bit (and maybe reconsiders some life choices, like not becoming an Amish leatherworker), and decides to restore from the single-disk backup that's sitting right there (it might be a few days old or whatever, but they decide it's OK).

And that's sounding pretty good, except: With most filesystems, we have no way to tell if that backup drive is suffering from bitrot. It contains only presumably good data. But that presumed-good data is soon to become the golden sample from which all future backups are made: When that backup has rotten data, then it silently poisons the present system and all future backups of that system.

If that offline disk instead uses ZFS, then the system detects and reports the rot condition automatically upon restoration -- just in the normal course of reading the disk, because that's how ZFS do. This allows the user to make informed decisions that are based on facts instead of blind trust.

With ZFS, nothing is silently poisoned.

Better than RAIDZ1, which is what you suggested, in the sense it's more reliable. I didn't say anything about not using ZFS.

I never mentioned RAIDZ1 (and neither did you, until just now).

It doesn't matter but you can just go look.

I had to check for data integrity due to a recent system switch, and was surprised not to find any bitrot after 4y+.

It took ages to compute and verify those hashes between different disks. Certainly an inconvenience.

I am not sure a NAS is really the right solution for smaller data sets. An SSD for quick hashing and a set of N hashed cold storage HDDs - N depends on your appetite for risk - will do.

I've hosted my own data for twenty something years - and bitrot occurs but it is basically caused by two things.

1) Randomness <- this is rare 2) HW-failures <- much more common

So if you catch hw-failures early you can live a long life with very little bitrot... Little =! none so zfs is really great.

Don’t get me wrong: IMHO a ZFS mirror setup sounds very tempting, but its strength lie in active data storage. Due to the rarity of bitrot I would argue it can be replaced with manual file hashing (and replacing, if needed) and used in cold storage mode for months.

What worries me more than bitrot is that consumer disks (with enclosure, SWR) do not give access to SMART values over USB via smartctl. Disk failures are real and have strong impact on available data redundancy.

Data storage activities are an exercise in paranoia management: What is truly critical data, what can be replaced, what are the failure points in my strategy?

There's no worse backup system than that which is sufficiently-tedious and complex that it never gets used, except maybe the one that is so poorly documented that it cannot be used.

With ZFS, the hashing happens at every write and the checking happens at every read. It's a built-in. (Sure, it's possible to re-implement the features of ZFS, but why bother? It exists, it works, and it's documented.)

Paranoia? Absolutely. If the disk can't be trusted (as it clearly cannot be -- the only certainty with a hard drive is that it must fail), then how can it be trusted to self-report that it is has issues? ZFS catches problems that the disks (themselves inscrutable black boxes) may or may not ever make mention of.

But even then: Anecdotally, I've got a couple of permanently-USB-connected drives attached to the system I'm writing this on. One is a WD Elements drive that I bought a few years ago, and the other is a rather old, small Intel SSD that I use as scratch space with a boring literally-off-the-shelf-at-best-buy USB-SATA adapter.

And they each report a bevy of stats with smartctl, if a person's paranoia steers them to look that way. SMART seems to work just fine with them.

(Perhaps-amusingly, according to SMART-reported stats, I've stuffed many, many terabytes through those devices. The Intel SSD in particular is at ~95TBW. There's a popular notion that using USB like this sure to bring forth Ghostbusters-level mass hysteria, especially in conjunction with such filesystems as ZFS. But because of ZFS, I can say with reasonable certainty that neither drive has ever produced a single data error. The whole contrivance is therefore verified to work just fine [for now, of course]. I would have a lot less certainty of that status if I were using a more-common filesystem.)

I agree about manual file hashing. For data that rarely changes it also has some benefits.

Some time ago, I ended up writing a couple of scripts for managing that kind of checksum files: https://github.com/kalaksi/checksumfile-tools

This works great although I should really do step 4 :)