Today ... has not gone to plan.
Oddly, I think it started to go wrong on the way home, rather than while I was still at work (although I got no words written today on account of having to fight a series of administrative fires). I was happily cycling home in the dark, lights and high-vis on, everything more or less OK, when my rear wheel started to vibrate interestingly. I've been here before. I pulled over. Rear tyre, completely flat. Arse.
So I pushed my ironically-named pushbike home, and locked it to the mooring loop, as usual, and took the rear wheel off and brought it in with me, to fix the puncture. Since I'd just cycled/walked home, I took five minutes to spod on the 'net, and my music collection disappeared. The player stopped in the middle of a song, and when I went to check it there were no songs. No songs, no directory, nothing.
Troubling. I could swear they were here only a minute ago.
So, it seems the highly reliable RAID I built a couple of years back to store my stuff ... isn't so highly reliable any more. It's a 3 disk array, which means it can lose any one disk without losing any data. It's dropped one particular disk a couple of times recently, both times pretty soon after street-wide power cuts, so I figured it was a little tired, and planned to make backups etc. Real Soon. Unfortunately, now it's dropped that disk, and its brother... Array Failure.
RAID arrays spread the contents of a file across multiple disks to increase redundancy, so I don't even have a few of my many files intact on the remaining disk: I just have bits of the vast majority of my files. Those bits don't add up to a whole, in most cases. None of my skills with Linux seemed to be working, and I couldn't get it to rebuild.
Then, dwm turned up on IRC, and guided me through a six-hour rebuild that hasn't quite finished yet, but looks like it's going to work. Once it's done, I need to pull important things off that drive, then see what happens, I guess. I'm not quite sure what's wrong with the machine, but signs point to one of the SATA controllers going bad, which would be very annoying.
So, bit of a wakeup call there. A lot of my data is pretty insignificant, but I'd miss it if it were gone. So I must implement a Proper Backup Strategy in the soon-after...
I eventually solved the bicycle problem too. Patched the tube, went round the tyre to find the cause, found a 10mm shard of road-grit that had gone right through the tyre wall and slashed the tube. Yet another reason to hate snowy weather...
Oh well. The world turns, things happen, or don't, according to their own inscrutable timetable, and we all get slowly older. This is the way of things.
2010/01/19 - Craig (09:16)
We all do it, we all know better and we all still make the mistakes.
While my system itself is fairly well backed up, it is not as rigid as I would like and my disks are older than I would like. My websites are backed up for most part, but in a messy kinda of way and the databases have no decent kind of backup.
Much as I am one for complaining at others for their lack of backup routines (I remember that password file dan...) perhaps you are reminding me that I should devote some time to my own backups too this week.
Hope the rebuild works and holds and hope a backup routine follows. (rsync to icy box?)
2010/01/19 - Pewterfish (15:48)
Mmm, backups are good. The rebuild succeeded at 4AM this morning (handheld all the way by yours truly, although dwm quite understandably went to bed once we were into the fsck proper, as I could handle if from there), and I'm pulling backups of the important things as we speak (passwords, photos, music etc.). I'm thinking pretty strongly of building myself a BIG fileserver and using something like backintime (http://backintime.le-web.org/) to mirror /home to it. Additionally, I'm looking favourably at rsync.net (http://rsync.net/) - secure networked backup to geographically distributed servers, from 80 cents per gig per month? Sounds good to me.
I guess we'll see. This latest fuckup has taught me, and I shall be advising others, to take care of {my,your} backups!
2010/01/19 - Torkell (21:55)
Eeep... Murphy really has it in for you..
One of the many things on my todo list is sorting out real offsite backups. My backups at the moment will survive any one disk failing and also will let me recover from accidental deletions, but that still leaves me vulnerable if the whole machine gets toasted. Ideally I want offsite backups. I have ideas for how to arrange such (ranging from keeping an external harddisk at work to running a server at parents or elsewhere) and most of the parts needed for any idea, but I just need to actually sit down and sort it out.
Metis has no backups whatsoever, so it's just as well that I don't use it as a primary store for anything important. Need to buy some more disks for it and set up some kind of backup system. One thing I do try at least to do is maintain a distinction between stuff that should be backed up and stuff that's less important (and make sure that the former is actually backed up), though it's still downright annoying when any data vanishes.
2010/01/20 - Pewterfish (11:33)
I didn't even say anything about the way my laptop kept crashing while I was trying to recover the disks over SSH... Murphy is an asshole :)
2010/01/19 - Craig (22:59)
Gentlemen, we so need to start an offsite backup syndicate....
Have a core of upto 10gig or something per person which is replicated overnight to a selection of machines. rsync or alike is clearly the only sane option otherwise myself and Thomas would never have any bandwidth.
Last I looked you two both have the machines for that and on my purchase list for a couple of months time is an always on server beginning at 1.5TB and moving up to 3TB soon after.
Actually, something like http://rsyncrypto.lingnu.com looks like it might do the trick for security (not to mention ssh).
Not quite sure of the details of how all this would work.. but thoughts?
2010/01/20 - Pewterfish (11:42)
Hmm. rsyncrypto bothers me slightly: some paranoid corner of my brain is screaming about known-plaintext attacks, but I think it's getting overexcited. I think I'd probably go with a set of small TrueCrypt volumes, personally, and upload on change, with each one being dedicated to storing a certain thing (50MB one containing passwords and financial data, several 100MB ones holding photos, etc).
I'm up for it, though I might be a bit tight for storage until I get the new FS up (3TB of space, but to be built around April due to component costs). I can probably dig 20GB of storage out of a machine somewhere in this network, and I like the idea of an online backup syndicate.
Craig: Any thoughts on cameras? :)
2010/01/20 - Craig (17:07)
No thoughts on cameras as yet. Did have a little look last night, but wasn't happy I was finding things that fitted the spec. Will have a sit down tonight and have a proper look.
As for backups and alike. I think I agree that starting smaller is better especially as I can't think of that much data that I want replicated *that* much. I already make paranoid backups and as soon as I move to a new case my backup drive will be removable in under a second which is my only real issue with my backups (they are on one site).
As for time, I don't plan on having a server for this kind of thing for a couple of months given other items are higher up my list, so I don't think we are short of time to work out how we are going to do this anyway. I'm planning on going to an all singing all dancing NAS box as I don't trust my unix skills with managing RAID etc... that and it comes with a set of tools (web based) that would take me years to find, collate and integrate to give me anything like the easy to use functionality.
I am actually (and strangely) swimming in storage, but only because I replace my old 160GB system drive with a new 1TB one...