External storage solutions? (Hard drives, SD cards, etc)

  • 🇵🇦 Nuestro primer dominio localizado está en español en kiwifarms.pa. Our first localized domain is on Spanish on kiwifarms.pa.
  • Want to keep track of this thread?
    Accounts can bookmark posts, watch threads for updates, and jump back to where you stopped reading.
    Create account
So that'd be about $1/month to store everything you have and then $2 to recover it.
And Azure and more recently Google Cloud also have cheap archival offerings.

But the biggest cost for retrieval is in "egress," getting your data out of their cloud back to your system at home, they're all quite expensive. AWS is $0.09/GiB, so your talking about $92 to get back a full TiB. Which works for major disaster recovery when a lot of money is flying around as you make a big insurance claim and replace a lot of things, but less so for failure of a single hard drive.
 
Última edición:
And Azure and more recently Google Cloud also have cheap archival offerings.

But the biggest cost for retrieval is in "egress," getting your data out of their cloud back to your system at home, they're all quite expensive. AWS is $0.09/GiB, so your talking about $92 to get back a full TiB. Which works for major disaster recovery when a lot of money is flying around as you make a big insurance claim and replace a lot of things, but less so for failure of a single hard drive.
Added that into my post just so it's consolidated. Big slip on my part.
That said, I think the cost is reasonable if I understand the files being talked about.
 
I do have another hard drive, though it's an older one, as I mentioned in the previous post. I'm rsyncing the two as we speak.

I suppose I can just keep two copies rsync'd every once in a while, and buy a new one when one inevitably shits itself. I'd have to have an extreme bout of bad luck for both to peter out within a few days of each other.
You should use something like CrystalDiskMark and check in on your drives every once in a while. Thankfully I did that recently and saw that a drive was marked yellow, so I copied everything over to a new one and thank god, that drive soon lost 200GB of data after windows forced a diskcheck. I've had two hard drives fail me, ever*, and one was a portable and the other one internal, both within 1-2 months of each other. Things can break down quickly.

*Not counting DeskStars...
 
You should use something like CrystalDiskMark and check in on your drives every once in a while. Thankfully I did that recently and saw that a drive was marked yellow, so I copied everything over to a new one and thank god, that drive soon lost 200GB of data after windows forced a diskcheck. I've had two hard drives fail me, ever*, and one was a portable and the other one internal, both within 1-2 months of each other. Things can break down quickly.

*Not counting DeskStars...
UnRAID has something called "PreClear" tool that does all the disk checking BEFORE You insert it into your array. Scrubs the disk bare, checks the drive for errors, levels it for use, then sends you a nice message "Disk is ready to go"
 
For actual archival storage - Accept nothing less than M-Disc BDXL.
Yes, they're pricey for 100Gb disks, but fuckit. The startup cost is significantly smaller than tape. For the cost of a good enterprise tape drive I can put together a couple redundant PCs, and source enough spare parts, and spare drives to last me a literal lifetime.
I run a bunch of Hitachi Enterprise HDDs in the home server, anything actually important and irreplaceable is stored on BD.

E: Any cloud recommendations are silly. The amount of data loss and service interruptions that happen in that space is much more significant than people realize. But I'm jaded due to the nature of my profession. If etching my data onto physical stone was even somewhat feasible in a timely manner I'd invest in multiple hammer and chisel sets.
 
UnRAID has something called "PreClear" tool that does all the disk checking BEFORE You insert it into your array. Scrubs the disk bare, checks the drive for errors, levels it for use, then sends you a nice message "Disk is ready to go"
Yes, but during the life of the disk it is good to check in on it once in a while and Crystal Disk Mark checks the current SMART values and presents the state of it in an easy way. Don't download the inexplicable anime version of the program though.
 
For actual archival storage - Accept nothing less than M-Disc BDXL.
Yes, they're pricey for 100Gb disks, but fuckit. The startup cost is significantly smaller than tape. For the cost of a good enterprise tape drive I can put together a couple redundant PCs, and source enough spare parts, and spare drives to last me a literal lifetime.
I run a bunch of Hitachi Enterprise HDDs in the home server, anything actually important and irreplaceable is stored on BD.

E: Any cloud recommendations are silly. The amount of data loss and service interruptions that happen in that space is much more significant than people realize. But I'm jaded due to the nature of my profession. If etching my data onto physical stone was even somewhat feasible in a timely manner I'd invest in multiple hammer and chisel sets.
Different strokes 'n all. As long as all of your eggs aren't in one basket you should always be able to recover data before the next disaster.
 
You should use something like CrystalDiskMark and check in on your drives every once in a while.
Very old, decade and a half data that does not necessarily apply to today's drives from Google said S.M.A.R.T. only gave warning of impending disk failure half of time time. Ideally look for software that constantly checks reported disk health and alerts you if there's a problem.

In the Linux and probably BSD worlds and perhaps ported to macOS and/or Windows there's the smartd daemon from smartmontools, which also gives you a neat command line interface to this data including the ability to do self-checks, although avoid doing the long ones very often because today's disks have very limited official total amount of data you can read or write per year.

Just confirmed for Seagate's best and biggest five year warranty enterprise drives they still have less than 550TB per year, a limit they've had for years no matter the capacity of the drive, but one for some time you have to read the product manual to find out. Consumer drives typically have significantly lower official limits, and three or fewer years warranties speak for themselves.
For actual archival storage - Accept nothing less than M-Disc BDXL.
M-DISC the company is half a decade dead.

Once upon a time, and maybe still today their one layer DVDs were supposed to be very good, but I've never been able to find out if that technology is used in multi-layer BD's with that marketing label, and I have serious doubts it would work for the semi-transparent layers.

And last time I checked a few years ago these discs were made by CMC in India, all the good companies dropped out of the consumer market, one or two Japanese ones still make media using this technology for enterprises but it's not compatible with the consumer drives we can easily buy.
E: Any cloud recommendations are silly. The amount of data loss and service interruptions that happen in that space is much more significant than people realize.
Who gives a damn about service interruptions for backup and emergency, very expensive retrieval?

I have heard AWS S3 isn't as reliable as they claim, and the whole company is so poorly run it would be crazy to put all your eggs in that one basket, sooner or latter they might have a catastrophic failure from humans or code. The advice all sane people give is to use two or more completely different cloud vendors. So maybe AWS and Microsoft Azure so your total monthly at rest cost is $3/TiB.
 
Última edición:
Once upon a time, and maybe still today their one layer DVDs were supposed to be very good, but I've never been able to find out if that technology is used in multi-layer BD's with that marketing label, and I have serious doubts it would work for the semi-transparent layers.
I have no-brand DVDs I used to burn huge amounts of Xbox games, movies and PC releases back when that was relevant. I bought the cheapest ones in bulk via mailorder from shady shops that only sold these kinds of things. They don't even have any markings, the face side is just blank. They were dirt cheap, I stored them on spindles in a storage unit in boxes filled with other crap and all of them still works which surprised me.

This is what they look like and what they were stored like.
zDVD.jpg

When I started to pull things off of them I remembered how slooooow optical media was compared to everything we have now.
 
Última edición:
Tape drives. External USB disks are shit in general.
In case tapes are too slow and information is really important - build your own home NAS server (just watch out for SMR drives: you do not want them in RAIDs of any kind. For NAS OS use TrueNAS, for example. Do NOT buy WD My Book NASes and similar ready-made shit which is slow and may delete your files due to buggy update, build your own like a real man.
There are disks that are certified to last for 300 years before any degradation. They are called M-Disks and have jo to 100GB of storage capacity. Note that are write ONCE so don't mess it up.
 
.M-DISC the company is half a decade dead.

Once upon a time, and maybe still today their one layer DVDs were supposed to be very good, but I've never been able to find out if that technology is used in multi-layer BD's with that marketing label, and I have serious doubts it would work for the semi-transparent layers.

And last time I checked a few years ago these discs were made by CMC in India, all the good companies dropped out of the consumer market, one or two Japanese ones still make media using this technology for enterprises but it's not compatible with the consumer drives we can easily buy.Who gives a damn about service interruptions for backup and emergency, very expensive retrieval?

The last pack of Verbatim branded BD M Disks I got are jap made. The "Life Series" discs that where sold in big box stores are made by CMC.
And to be honest I really don't think there is a difference between M-disc certified BD-r and regular BD-r. The big difference was with the DVDs.

When I started to pull things off of them I remembered how slooooow optical media was compared to everything we have now.
Thats why I only use it for cold storage. Burn it, store it, and forget it until needed. Something I would need frequent access to I would put on NAS. Cloud is just some one else's NAS you pay to use. And if you get sick or your billing gets fucked up for a few months your data is deleted. The disc's have the advantage of you can literally forget about them in a drawer for 10 years like you did and they will still there be there when you remember them.

Since OP never gave us any more details about his situation this discussion is pointless.
Would I try and burn 30TB to 25gig BD's? No that would be dumb.
Would I pay AWS every month to keep 600megs data safe? No
Would 3TB of cloud work out better for a guy compared to an onsite NAS if he only has a 3m/768k DSL line? Nope.
What about Cloud vs a big NAS for a nomadic worker?
 
but how do you store any of those solution in your anus in case the Feds are at the door? SD cards, lots of them in a condom are the only solution.
But it takes a lot of training to be able to crush them instantly in case of emergency.
 
last time I checked a few years ago these discs were made by CMC in India, ..., one or two Japanese ones still make media using this technology for enterprises but it's not compatible with the consumer drives we can easily buy.

Considering that my little Samsung external BD burner is compatible with 100gb BDXL (M-)disks, and Verbatim apparently still produces their media in Japan, you appear to be incorrect on every single front there. I usually wouldn't argue the point, but I literally just got the reason to unpack the drive and media the other day and actually get some archival burning done.

Cloud Storage is a reasonable proposition only for the short/medium term storage, and we're talking < 5 years here. And you will be doing proper multiple-provider redundancy if you actually need it to be reliable. But then you're depending on your crappy consumer internet uplink to have any sort of actual reliability, which by default it won't. The claimed howevermany9s uptimes that amazon has is already bullshit, because they were down multiple times for more than an hour in December alone.
 
Última edición:
I will repost a post of mine that should've been here because I have been thinking about possibly a home NAS for a while now.

since we're on the topic of backup solutions, I've had the idea for a while to buy a consumer grade NAS with maybe 6 TB storage and two drives mirrored. (I estimate about $400 total) Most of my files is raw footage for youtube, computer backups, and 20 years of family photos/videos. I also thought of a poor man's "NAS" of literally just a RPi hooked up to an external drive. Which sounds more reasonable? Currently I'm just manually backing up compressed tarballs to an external drive, but if I want to free up space on my computer, I need to have the video files existing redundantly somewhere (I've had a past external drive almost catastrophically fail which means ideally the external drive which has files from a bunch of different places itself has a backup)
edit: of course I could just store it in a aws bucket but where's the fun in that

and no I'm not using 300 fucking DVDs (which I dont trust for data integrity and are painfully slow) or magnetic tapes (which are a good option for companies with many many terabytes of data but overkill for a consumer.)

I don't like how the consumer level NAS bay itself is $300. ive also considered just a cronjob that periodically rsyncs two disks as backup.
 
I will repost a post of mine that should've been here because I have been thinking about possibly a home NAS for a while now.



and no I'm not using 300 fucking DVDs (which I dont trust for data integrity and are painfully slow) or magnetic tapes (which are a good option for companies with many many terabytes of data but overkill for a consumer.)

I don't like how the consumer level NAS bay itself is $300. ive also considered just a cronjob that periodically rsyncs two disks as backup.
For all the hate they get, Synology NAS offerings in the 4-6 drive space are pretty good. Not amazing but they have loads of support and are pretty much plug and play.

Rolling your own WILL be cheaper and will give you unlimited customization options BUT if you fuck it up you're SoL.
 
For all the hate they get, Synology NAS offerings in the 4-6 drive space are pretty good. Not amazing but they have loads of support and are pretty much plug and play.

Rolling your own WILL be cheaper and will give you unlimited customization options BUT if you fuck it up you're SoL.
I want to roll my own because it's more fun. It's not as fun to just have everything in an amazon bucket or dropbox or gdrive.
Do you think this is the best option for like 1-10 TB range of home storage? I'm cheap and dont know if it's worth spending $250 on a NAS bay
 
I want to roll my own because it's more fun. It's not as fun to just have everything in an amazon bucket or dropbox or gdrive.
Do you think this is the best option for like 1-10 TB range of home storage? I'm cheap and dont know if it's worth spending $250 on a NAS bay
If you want to roll your own and have more reliable disks then skip SATA and go for SAS. More expensive and you will need a SAS card, but the upside is that if you have a SAS card you will likely have hardware RAID capabilities. SAS is compatible with SATA so you don't have to get those drives immediately, and you will still have raid. Buy a used minitower office computer (dell/hp/fujitsu/lenovo) for 100-150 bucks and put everything into that, install linux, hide it somewhere, run it headless, turn it on(wake on lan) and VNC into it when necessary.
That's the hardware covered, it's not that expensive so far.

For software you obviously need to get IBM Tivoli Storage Manager.
 
You should use something like CrystalDiskMark and check in on your drives every once in a while. Thankfully I did that recently and saw that a drive was marked yellow, so I copied everything over to a new one and thank god, that drive soon lost 200GB of data after windows forced a diskcheck. I've had two hard drives fail me, ever*, and one was a portable and the other one internal, both within 1-2 months of each other. Things can break down quickly.

*Not counting DeskStars...
I got so lucky. None of my Deskstars I owned went bad on me.
Because I use a ICYDOCK system on my computer that has 4, 5.25" bays I am able to swap out SSD's and HDD's.

Yea I know that HDD's are uncool but you're looking at a guy who was doing tech since many was only a gleam in their daddy's eyes. HDD's now are CHEAP for storage, so making copies as well as storage is very easy to protect your data. Hot Swapping is such a great thing.
 
Atrás
Top Abajo