2021-06-27-backups-update-internet-celebrity.gmi (4703B)
1 # An Update on Bad Backups and Internet Celebrity 2 3 My recent post on the Great Alpine Migration Tragedy of 2021 garnered some serious attention! I got a number of emails, a whole host of feedback on Lobste.rs, and over 100 points on Hacker News! As I understand it, that means I'm now a top internet celebrity (and eligible for the 10 KB Club). 4 5 Still, I must remain humble even as I catapult toward digital infamy and thus I am here to respond to feedback, answer questions, and talk about what I've done to prevent something like this from happening again. 6 7 Before we begin, for those who haven't already, read the story of the Great Alpine Migration Tragedy: 8 => /log/wrong-way-to-switch-server-os.md over Gemini 9 => https://lobste.rs/s/ofcfbq/wrong_way_switch_operating_systems_on on Lobste.rs 10 => https://news.ycombinator.com/item?id=27562613 on Hacker News 11 12 ## Useful Tips from Strangers 13 14 ### Ctrl-C is Not a Debugging Tool 15 16 That's a good point: I've been advised to update my methods. Several folks pointed me towards strace. Sending the SIGUSR1 or SIGINFO signal to Tarsnap would also work. The -v flag could have been helpful as well. 17 18 => https://jvns.ca/blog/2021/04/03/what-problems-do-people-solve-with-strace/ Also see: What problems do people solve with strace? 19 20 ### Don't Wipe Prematurely 21 22 A number of folks were confused as to why I didn't keep the old machine around during the transition. Others detailed their own migration strategies. In general, it seems that folks agree you should keep around your old machine until you're totally certain everything is up and running in your new setup. 23 24 As I mention briefly in the article, I've actually done this in the past myself! This time, however, there was no second server. Everything was done in-place. I cannot change this, as it occurred in the past. In the future I will avoid this kind of migration. 25 26 ### A Restore System 27 28 A number of people took issue with my take-aways from the experience. See: 29 30 > Trying to address a lack of confidence by increasing the backup 31 > frequency doesn't make sense. The backup frequency is the most trivial 32 > thing to adjust and doesn't address deeper issues, like the fact that 33 > you need to dump/restore databases properly and shouldn't copy files 34 > from a live database. 35 => https://news.ycombinator.com/item?id=27596126 by klodolph 36 37 I'll start by saying that that increasing backup frequency from zero to anything greater than zero does, in fact, address a significant issue. I now have backups where there were none before. 38 39 Still, the point stands. Without quality backups, and quality restore strategies, the backups are not as useful as they could be. 40 41 ### A Little Bit of Gold and a Pager 42 43 There was some fun memeing in the Hacker News comments in response to me being a teenager: 44 45 > I know whom I won't hire for my company IT or devops or whatnot. 46 => https://news.ycombinator.com/item?id=27597216 by wildmanx 47 48 Several people have a sense of humour: 49 50 > Good idea to let him graduate from highschool before hiring him! 51 => https://news.ycombinator.com/item?id=27597698 by nick__m 52 53 I'll add for further memery that in addition to the stress of my entire server infrastructure burning, I had to deal with passing the behind-the-wheel driver's test. I've now been a licensed driver for about a week and a half. Coincidentally, I got my server back running the same day I passed the test. 54 55 ## How I'm Doing Backups 56 57 I moved all the data in my self-hosting setup into a single directory. The files were previously all lying about my home directory, which is what led to the tragic "forgetting about .env amongst all my dotfiles" incident. 58 59 I set up a cronjob that runs a script daily, set to back up the self hosting directory. It currently keeps a buffer of 31 daily and 12 monthly backups, and keeps around yearly backups forever. It's been running for a couple days now, and it's all been smooth sailing so far. 60 61 => https://github.com/alexjurkiewicz/acts/ acts: Another Calendar-based Tarsnap Script 62 63 ## To Infinity and Beyond 64 65 I'd also like to say thanks to a number of people who were especially kind, and made this a fun experience: 66 67 1. Graham Percival, an employee at Tarsnap Backup Inc., reached out to me directly – we had a fun and interesting email exchange. 68 2. hannu was the first to write the later much-echoed sentiment that reflecting on one's mistakes is a useful and laudable thing to do. 69 => https://lobste.rs/s/ofcfbq/wrong_way_switch_operating_systems_on#c_b0zqnt hannu on Lobste.rs 70 71 I'm going to continue iterating my self-hosting setup. I've still got a couple new services I might want to throw up there. I also can't forget about the dedicated "self-hosting" page I promised...