figbert.com-website

[ACTIVE] the website and home of figbert on the clearnet
git clone git://git.figbert.com/figbert.com-website.git
Log | Files | Refs | README | LICENSE

backups-update-internet-celebrity.md (5088B)


      1 +++
      2 title = "An Update on Bad Backups and Internet Celebrity"
      3 date = 2021-06-27
      4 updated = 2022-06-15
      5 +++
      6 
      7 My recent post on the [Great Alpine Migration Tragedy] of 2021 garnered
      8 some serious attention! I got a number of emails, a whole host of
      9 feedback on [Lobste.rs], and over **100 points** on [Hacker News]! As I
     10 understand it, that means I'm now a top internet celebrity (and eligible
     11 for the [10 KB Club]). Still, I must remain humble even as I catapult
     12 toward digital infamy and thus I am here to respond to feedback, answer
     13 questions, and talk about what I've done to prevent something like this
     14 from happening again.
     15 
     16 <!-- more -->
     17 
     18 ## Useful Tips from Strangers
     19 
     20 ### Ctrl-C is Not a Debugging Tool
     21 
     22 That's a good point: I've been advised to update my methods. Several
     23 folks pointed me towards [`strace`]. Sending the `SIGUSR1` or `SIGINFO`
     24 signal to Tarsnap would also work. The `-v` flag could have been
     25 helpful as well.
     26 
     27 ### Don't Wipe Prematurely
     28 
     29 A number of folks were confused as to why I didn't keep the [old
     30 machine] around during the transition. Others detailed their own
     31 [migration strategies][klodolph]. In general, it seems that folks agree
     32 you should keep around your old machine until you're totally certain
     33 everything is up and running in your new setup.
     34 
     35 As I mention briefly in the article, I've actually done this in the past
     36 myself! This time, however, there was no second server. Everything was
     37 done in-place. I cannot change this, as it occurred in the past. In the
     38 future I will avoid this kind of migration.
     39 
     40 ### A Restore System
     41 
     42 A number of people took issue with my take-aways from the experience.
     43 See:
     44 
     45 > Trying to address a lack of confidence by increasing the backup
     46 > frequency doesn't make sense. The backup frequency is the most trivial
     47 > thing to adjust and doesn't address deeper issues, like the fact that
     48 > you need to dump/restore databases properly and shouldn't copy files
     49 > from a live database.
     50 >
     51 > by [klodolph]
     52 
     53 I'll start by saying that that increasing backup frequency from zero to
     54 anything greater than zero does, in fact, address a significant issue. I
     55 now have backups where there were none before.
     56 
     57 Still, the point stands. Without quality backups, and quality restore
     58 strategies, the backups are not as useful as they could be.
     59 
     60 ### A Little Bit of Gold and a Pager
     61 
     62 There was some fun memeing in the Hacker News comments in response to me
     63 being a teenager. Specifically, wildmanx said that they [wouldn't hire
     64 me for their "company IT or devops or whatnot."][hire]. Several people
     65 have a sense of humour:
     66 
     67 > Good idea to let him graduate from highschool before hiring him!
     68 >
     69 > by [nick__m]
     70 
     71 I'll add for further memery that in addition to the stress of my entire
     72 server infrastructure burning, I had to deal with passing the
     73 behind-the-wheel driver's test. I've now been a licensed driver for
     74 about a week and a half. Coincidentally, I got my server back running
     75 the same day I passed the test.
     76 
     77 ## How I'm Doing Backups
     78 
     79 I moved all the data in my self-hosting setup into a single directory.
     80 The files were previously all lying about my home directory, which is
     81 what led to the tragic "forgetting about `.env` amongst all my dotfiles"
     82 incident.
     83 
     84 I set up a cronjob that runs the backup script [`acts`] daily, set to
     85 back up the self hosting directory. It currently keeps a buffer of 31
     86 daily and 12 monthly backups, and keeps around yearly backups forever.
     87 It's been running for a couple days now, and it's all been smooth
     88 sailing so far.
     89 
     90 ## To Infinity and Beyond
     91 
     92 I'd also like to say thanks to a number of people who were especially
     93 kind, and made this a fun experience:
     94 1. Graham Percival, an employee at Tarsnap Backup Inc., reached out to
     95    me directly – we had a fun and interesting email exchange.
     96 2. [hannu] was the first to write the later much-echoed sentiment that
     97    reflecting on one's mistakes is a useful and laudable thing to do
     98    (see also [abraae] and [argomo]).
     99 
    100 I'm going to continue iterating my self-hosting setup. I've still got a
    101 couple [new services] I might want to throw up there. I also can't
    102 forget about that [dedicated page] I promised...
    103 
    104 [Great Alpine Migration Tragedy]: @/posts/wrong-way-to-switch-server-os.md
    105 [Lobste.rs]: https://lobste.rs/s/ofcfbq/wrong_way_switch_operating_systems_on
    106 [Hacker News]: https://news.ycombinator.com/item?id=27562613
    107 [10 KB Club]: https://10kbclub.com
    108 [`strace`]: https://jvns.ca/blog/2021/04/03/what-problems-do-people-solve-with-strace/
    109 [old machine]: https://lobste.rs/s/ofcfbq/wrong_way_switch_operating_systems_on#c_sfwzdq
    110 [klodolph]: https://news.ycombinator.com/item?id=27596126
    111 [hire]: https://news.ycombinator.com/item?id=27597216
    112 [nick__m]: https://news.ycombinator.com/item?id=27597698
    113 [`acts`]: https://github.com/alexjurkiewicz/acts/
    114 [hannu]: https://lobste.rs/s/ofcfbq/wrong_way_switch_operating_systems_on#c_b0zqnt
    115 [abraae]: https://news.ycombinator.com/item?id=27597436
    116 [argomo]: https://news.ycombinator.com/item?id=27598224
    117 [new services]: https://www.firefly-iii.org
    118 [dedicated page]: @/posts/wrong-way-to-switch-server-os.md#bye-bye