backups-update-internet-celebrity.md (5110B)
1 +++ 2 title = "An Update on Bad Backups and Internet Celebrity" 3 date = 2021-06-27 4 updated = 2022-06-15 5 [extra] 6 type = "post" 7 +++ 8 9 My recent post on the [Great Alpine Migration Tragedy] of 2021 garnered 10 some serious attention! I got a number of emails, a whole host of 11 feedback on [Lobste.rs], and over **100 points** on [Hacker News]! As I 12 understand it, that means I'm now a top internet celebrity (and eligible 13 for the [10 KB Club]). Still, I must remain humble even as I catapult 14 toward digital infamy and thus I am here to respond to feedback, answer 15 questions, and talk about what I've done to prevent something like this 16 from happening again. 17 18 <!-- more --> 19 20 ## Useful Tips from Strangers 21 22 ### Ctrl-C is Not a Debugging Tool 23 24 That's a good point: I've been advised to update my methods. Several 25 folks pointed me towards [`strace`]. Sending the `SIGUSR1` or `SIGINFO` 26 signal to Tarsnap would also work. The `-v` flag could have been 27 helpful as well. 28 29 ### Don't Wipe Prematurely 30 31 A number of folks were confused as to why I didn't keep the [old 32 machine] around during the transition. Others detailed their own 33 [migration strategies][klodolph]. In general, it seems that folks agree 34 you should keep around your old machine until you're totally certain 35 everything is up and running in your new setup. 36 37 As I mention briefly in the article, I've actually done this in the past 38 myself! This time, however, there was no second server. Everything was 39 done in-place. I cannot change this, as it occurred in the past. In the 40 future I will avoid this kind of migration. 41 42 ### A Restore System 43 44 A number of people took issue with my take-aways from the experience. 45 See: 46 47 > Trying to address a lack of confidence by increasing the backup 48 > frequency doesn't make sense. The backup frequency is the most trivial 49 > thing to adjust and doesn't address deeper issues, like the fact that 50 > you need to dump/restore databases properly and shouldn't copy files 51 > from a live database. 52 > 53 > by [klodolph] 54 55 I'll start by saying that that increasing backup frequency from zero to 56 anything greater than zero does, in fact, address a significant issue. I 57 now have backups where there were none before. 58 59 Still, the point stands. Without quality backups, and quality restore 60 strategies, the backups are not as useful as they could be. 61 62 ### A Little Bit of Gold and a Pager 63 64 There was some fun memeing in the Hacker News comments in response to me 65 being a teenager. Specifically, wildmanx said that they [wouldn't hire 66 me for their "company IT or devops or whatnot."][hire]. Several people 67 have a sense of humour: 68 69 > Good idea to let him graduate from highschool before hiring him! 70 > 71 > by [nick__m] 72 73 I'll add for further memery that in addition to the stress of my entire 74 server infrastructure burning, I had to deal with passing the 75 behind-the-wheel driver's test. I've now been a licensed driver for 76 about a week and a half. Coincidentally, I got my server back running 77 the same day I passed the test. 78 79 ## How I'm Doing Backups 80 81 I moved all the data in my self-hosting setup into a single directory. 82 The files were previously all lying about my home directory, which is 83 what led to the tragic "forgetting about `.env` amongst all my dotfiles" 84 incident. 85 86 I set up a cronjob that runs the backup script [`acts`] daily, set to 87 back up the self hosting directory. It currently keeps a buffer of 31 88 daily and 12 monthly backups, and keeps around yearly backups forever. 89 It's been running for a couple days now, and it's all been smooth 90 sailing so far. 91 92 ## To Infinity and Beyond 93 94 I'd also like to say thanks to a number of people who were especially 95 kind, and made this a fun experience: 96 1. Graham Percival, an employee at Tarsnap Backup Inc., reached out to 97 me directly – we had a fun and interesting email exchange. 98 2. [hannu] was the first to write the later much-echoed sentiment that 99 reflecting on one's mistakes is a useful and laudable thing to do 100 (see also [abraae] and [argomo]). 101 102 I'm going to continue iterating my self-hosting setup. I've still got a 103 couple [new services] I might want to throw up there. I also can't 104 forget about that [dedicated page] I promised... 105 106 [Great Alpine Migration Tragedy]: @/posts/wrong-way-to-switch-server-os.md 107 [Lobste.rs]: https://lobste.rs/s/ofcfbq/wrong_way_switch_operating_systems_on 108 [Hacker News]: https://news.ycombinator.com/item?id=27562613 109 [10 KB Club]: https://10kbclub.com 110 [`strace`]: https://jvns.ca/blog/2021/04/03/what-problems-do-people-solve-with-strace/ 111 [old machine]: https://lobste.rs/s/ofcfbq/wrong_way_switch_operating_systems_on#c_sfwzdq 112 [klodolph]: https://news.ycombinator.com/item?id=27596126 113 [hire]: https://news.ycombinator.com/item?id=27597216 114 [nick__m]: https://news.ycombinator.com/item?id=27597698 115 [`acts`]: https://github.com/alexjurkiewicz/acts/ 116 [hannu]: https://lobste.rs/s/ofcfbq/wrong_way_switch_operating_systems_on#c_b0zqnt 117 [abraae]: https://news.ycombinator.com/item?id=27597436 118 [argomo]: https://news.ycombinator.com/item?id=27598224 119 [new services]: https://www.firefly-iii.org 120 [dedicated page]: @/posts/wrong-way-to-switch-server-os.md#bye-bye