In the jungle, terrible jungle, a server died tonight

Hello everyone and thank you for clicking on this article 🙂

Today I'm going to tell you something dramatic:

On a beautiful spring afternoon, you're working on your server via your favorite terminal and all of a sudden boom. Nothing responds. You try to open the web sites you're managing, and nothing happens. It just keeps going until the browser times out. 🤦‍♂️

This is exactly what happened to me a few days ago, and I wouldn't recommend this experience to anyone.

So I quickly contacted my host's support team. They were very quick and efficient, but the problem was that they told me that the server had physically died (either the hard disk or the motherboard).

They then offered to open a new server free of charge as a replacement, which I accepted.

So now what?

First instinct: Thank your past self for having made backups.

Second thought: Make yourself a cup of coffee, as it's going to be a long evening.

Reinstalling a server

Once the new access has been validated, the first step is to install a distribution. I'm going back to a classic Ubuntu 20.04, which does the trick for everything I need. There's an installation and set-up delay, of course, but that's all right since everything's down anyway.

A few tens of minutes later, the server is up and configured. I connect via ssh, and the feeling of landing in a new world untouched by explorers quickly gives way to the task in hand. I install a few basic tools (nano, docker, nginx ...) needed to run my stack and perform a few operations.

Ok, that sounds good, now it's time to recover all the backups on this server and reassemble everything.

Backup recovery

Personally, I've been using duplicity since I started managing a hosting server myself. I find this tool practical and efficient, I have a routine that runs every day to make my backups and send them to another backup server (fortunately, isn't it? 😛 ).
So I send all my folders to my new server with

and the folders reappear in their intended location.

I then set up my nginx reverse proxy again to capture my domain names, which I redirect to the server's new IP. I get my 503 error page when trying to reach my domains, everything works (yes, I know it doesn't look like that 😀).

Come on, let's try a docker-compose up in one of the website folders, the one for Boréales Créations it's a showcase site it'll be the easiest to put back in place.

It works straight away, the site is up and accessible from the outside. No shit, thanks duplicity.

I'm moving on to more complex sites like Neartrip, but it's going well overall, everything's up, no loss of files or data. All I had to do was import the data again from the backup files.

Okay, in a few minutes/hours all the sites are up again, the longest being the time needed to compile the docker and import all the data.

Reconfiguring routines and services

Well, the main thing is done, my sites are back online. Okay, but that's not all there was on my server. In particular, there were my backup scripts and the corresponding cron jobs to run them every day. Problem, I didn't have a backup on that, it's a mistake on my part and I promise this time they're well preserved on my side 😀. That's how it is after a mistake you learn from it. In freelance life we're used to it 😊

Perfect, it's running, a few adjustments or improvements are made along the way, so I might as well make the most of it.

Lastly, I reinstalled Matomo to retrieve my site visit statistics. I think all's well, it's 01:00 in the morning, I'll be able to get a few hours' sleep despite today's drama.

Moral

Thank you duplicity! What do you mean I already said that? Well, to be more explicit, the moral is make backups. Of everything, everywhere, all the time. Even when you're thinking "If worst comes to worst, I'll do it again". No. Make the backup. You underestimate the satisfaction and relief of being able to recover everything and put it back in place in a minimum of time.

See you soon for another article with less suspense in it 😀