The New Oil

A few months ago, as my wife and I sat in the study doing out separate things, she suddenly asked if I noticed a certain smell in the room. Truthfully I had – a “chemically” smell is the only way I know how to describe it – but I have a notoriously bad sense of smell that sometimes plays tricks on me so as the scent came and went I simply assumed it was just another thing that was all in my head. However, once she spoke up I realized that I had been getting slightly light-headed, so we popped open a window, set up a fan, and moved to the living room to give the study time to air out. Afterward – the smell still very present but at least mixed with some fresh air – we went to hunt down the source of the issue. It ultimately ended up being a power strip that was going bad (did you guys know those things only have a shelf-life of a couple years? Be sure to replace your power strips periodically). Perhaps as a part of this problem or perhaps totally unrelated, it turned out that the power supply for my Jellyfin server had died (we initially incorrectly diagnosed this as the source of the problem). Never one to let an opportunity go to waste, I thought this might make a good blog post to share.

The Sting of Failure

Before I start, I should note that I don’t recommend self-hosting to everyone. Not everyone has the required skills or the time and resources to learn them, and some services are incredibly easy to screw up or too important to risk for some folks. That said, for those who do have the time, the hardware, and the low-enough threat model to risk mistakes, self-hosting can be an immensely rewarding and empowering experiment (not to mention some services – like Jellyfin – are unbelievably easy to set up). But like any new venture, it means you have to be ready to fail.

My server’s death was my first catastrophic failure in my ongoing self-hosting journey. I’ve definitely had issues before – the first few months of getting Nextcloud installed involved a weekly cycle of trial and error, starting over and fresh installs, and more. But this was the first time I lost data. The Nextcloud learning curve meant I knew to keep good backups and slowly grow my trust in the server. Jellyfin, in part due to the ease of setup, was up and running smoothly for months before this failure, so I was already well into trusting it. Thus, my first reaction was to get sad and frustrated.

For the record, I’m very much a “put your problems in perspective” kind of guy. You’ll rarely – if ever – find me losing my mind over the coffee shop being unexpectedly closed or my package getting lost in the mail. Of course, emotions cannot be controlled – only how we react to them can – so even though I knew my problems were very small in perspective (discussed next), I still couldn’t help but feel a pang of defeat and annoyance. I wondered if I was sure I wanted to keep self-hosting, or at very least if I didn’t want to just rent a VPS. I’m a strong believer that it’s okay to feel your feelings so long as you don’t let them run rampant. After putting out the metaphorical fires and clearing the smell out of the room, I spent the rest of the night ignoring the problem. I don’t remember exactly what I did, to be honest. Maybe poured a drink. Maybe watched a movie on another streaming service. Maybe played some video games. Regardless, I decided to just take the night off and worry about it the next day.

The Agony of Recovery

The good news is that Jellyfin contains basically no sensitive data. It’s all just movies and TV shows (I’ve since added a couple audiobooks) but I didn’t lose any backups, password vaults, documents, photos, etc. But it did mean I had to start all over. As most people know, you can’t just copy/paste an entire hard drive to a new one. You have to install the OS first – because the hardware is different and will contain different drivers and such – and then you have to reinstall the software, and then if there’s any sort of databases you have to migrate them over, etc. In the case of Jellyfin, neither myself or my wife was particularly attached to our show progress so we decided it was fine to simply reinstall everything, re-upload the existing content, and recreate the two user accounts. After dragging a different, unused device out of the closet and reinstalling Debian and Jellyfin, then came the next challenge: I didn’t want to re-rip every single piece of content we owned. That would make the process take at least twice as long. I knew the hard drive was still good, and I wanted to simply pull the perfectly good files off that and put them on the new device.

At the time this happened, I was experiencing my rocky transition from an old, dying device to a new one, and was currently without a Linux distro (I talked more about this in my System76 video, for those curious). Instead, I was using only Windows – unreliably, at that. I had to figure out how to mount the disk, pull all the files off, and transfer them to the new server. For the sake of word count, I’ll keep the details short, but suffice to say it took weeks of trying different possible solutions and ultimately I created a VM running Linux, attached both the drive and my external backup drive, and only then could I mount the old Jellyfin drive, transfer all the files directly to the backup disk, and then from there use FTP to send them back to the new server. All in all, this whole saga took several weeks (don’t forget I have a demanding day job) and was filled with a variety of emotions: sadness at my failure, frustration at not being able to watch my shows, annoyance at my device limitations and all the trial-and-error hoops I had to jump through. These days, my backup device contains a folder strictly for all my digitized media so that if this ever happens again, I don’t have to bother with the recovery phase.

The Joy of Learning

So why am I sharing this? To share my lessons, of course. As I’ve mentioned in past blogs, I do enjoy doing reviews and waxing philosophical about my opinions on various privacy topics, but I also enjoy personally sharing my experiences from the trenches. Believe it or not, I’m still semi-new to a lot of this stuff, too. I have a lot of theoretical knowledge but much of it I’ve never had the opportunity to test firsthand: I’m trying it out myself for the first time along with many of you and growing alongside a lot of readers. This applies mostly to advanced things like self-hosting multiple services, ghost addresses on my driver’s license, for the record. I’ve got the basics pretty mastered. (Also, of course, not all readers are new to this stuff. I know there’s some highly experienced folks out there and I thank you for your wisdom and advice.)

So what did I learn? First off, I learned failure can be depressing. That’s not really new, but life has a way of reminding us of that, especially when you tend to go through life with a high degree of confidence and competence. You’re not used to failing but it’s gonna happen eventually no matter what and when it does it really takes the wind out of your sails. That’s okay. Let it hurt. Take the night off. Vent. Relax. But don’t let it overtake you. Get back up on the horse tomorrow and try again. It’s important to remember that the struggle is (usually) worth it. For every week I’ve spent fighting with a server upgrade or rebuild or troubleshooting some other issue, I’ve had – on average – about 4-6 months (or more) of zero issues whatsoever. Pretty good ratio, in my opinion. It’s important to remember the benefit you get from whatever it is you’re doing and that the reward is worth the struggle. (Unless, of course, it’s not, but that’s a blog post for another time and a decision you shouldn’t make while you’re still deep in frustration or depression.) As the quote goes, “life is not about how many times you fall down. It’s about how many times you get back up.” The privacy journey in general – and life – is full of lots of falling down. But if you get back up you’ll learn something along the way and get better every time. I encourage you to always get back up. Oh, and keep good backups. I’m gonna go watch something on my Jellyfin server now.

You can find more recommended services and programs at TheNewOil.org, and you can find our other content across the web here or support our work in a variety of ways here.