As technology enthusiasts/developers, we want to play with the latest toys, download the latest Debian for testing, or install a web server on a Raspberry Pi.

But once we get into the corporate world, things don't work the same way.
To date, my servers run 5-6 services, some of which are crucial to my customers.

Because of this, I avoid downtime as much as possible and make sure I have a solid base that I don't need to touch every day.

(Not deploying security updates is not a solution).

This is one of the reasons why we've seen PaaS (Platform as a Service) solutions like Heroku, Clever Cloud... etc. emerge. These solutions allow you to focus solely on your application, and not on the infrastructure behind it.

But in my case, I've stuck with the good old server cluster, simply because it's not the same in terms of cost, and above all, I have no limits on creation.

For example, this application uses very recent technologies such as Typesense.

So, in order to develop the best possible applications, I need total freedom over the infra.
Just being able to manage web server headers can drastically change application performance.

Great, I've got total freedom, but I need to maintain the servers and above all not change distribution every two years (two years goes by fast).

Technologies like Docker are doing very well, with recent libs and binaries running in isolated containers, and the host running on stable, tested software.

But how can you park a stable, secure host for years on end?

10 years of security updates

In the Linux world, one company is well known for doing just that: RedHat.

In addition to this, Redhat provides hot kernel patching with kpatch, which allows you to patch the Linux kernel without rebooting.

A newcomer to OSes maintained for 10 years

Since January 2023, Ubuntu, via its Ubuntu Pro offer, has provided security patches for a period of 10 years. And the best part? It's free for up to 5 machines...

Well, it's free, but with a big limitation... more on that later...

Ubuntu Pro, when LTSs go from 5 to 10 years of updates...

Now we're getting to the heart of the matter...

Here's an overview of what Ubuntu Pro has to offer:

ESM" repositories for "Extended Security Maintenance", which offer updates for 10 years.
FIPS" packages, which seem to be certified packages, but I confess I haven't looked into them any further.
Hot kernel livepatching without rebooting, so we'll come back to that, as there are a few things you need to know.
USG to check that security standards are respected on the machine (audit tool)

How to get Ubuntu Pro

You'll need to go to the Ubuntu Pro page.

Further down the page, you'll see "free for personnal use".

Once you've made your choice, you'll have access to the Ubuntu Pro interface

Here you'll find your token, which you can activate with the "sudo pro attach" command. Note that the "pro" command is in the "ubuntu-advantage-tools" package.

This will activate the ESM repositories and give you access to livepatching... but be careful!

It's important to understand that the updated packages are those provided by Ubuntu and not those from third parties. For example, for docker, don't use the "docker-ce" repositories, but rather the version available in the Ubuntu repositories.

Limitations of the free version

Livepatching the free version puts you in the beta channel.
In other words, hot patches applied to the kernel, after being tested at Ubuntu, are deployed directly to your machines, which you'll agree is a bad idea for production use.

Updating the kernel with the free version

If you're using the free version, the best thing to do is to disable livepatching with the "sudo pro disable livepatch" command.

Once this has been done, please check that when the machine restarts, all the services it supports also restart without any particular problem.

Once this has been done, you can use the "unattended-upgrades" package to apply off-peak security patches.
To do this: "sudo apt install unattended-upgrades" then "sudo dpkg-reconfigure unattended-upgrades".

Once "unattended-upgrades" has been installed, I advise you to define when each action is to be performed, to avoid unpleasant surprises...

To do this, use the command "sudo systemctl edit apt-daily.timer" and add the following content:

[Timer] OnCalendar= OnCalendar=01:00 RandomizedDelaySec=5m

This will check repositories at 1:00 a.m. (with a 5-minute variation).

Next, use the command "sudo systemctl edit apt-daily-upgrade.timer" and add the following content:

[Timer] OnCalendar= OnCalendar=01:30 RandomizedDelaySec=0

At 1:30, if security patches need to be applied, they will be.

Now let's modify unattented-upgrades directly, to enable it to restart the server when necessary (usually when there are kernel updates):
To do this: "sudo nano /etc/apt/apt.conf.d/50unattended-upgrades".

Look for the line "Unattended-Upgrade::Automatic-Reboot" and change the value from "false" to "true".
To define when to reboot, modify the line "Unattended-Upgrade::Automatic-Reboot-Time" and set the value you want, in my case : Unattended-Upgrade::Automatic-Reboot-Time "02:00" for 2am.

Restart the "unattended-upgrades" service, and you're all set: "sudo systemctl restart unattended-upgrades".

To check that everything worked properly the next day, you can check the logs: "cat /var/log/unattended-upgrades/unattended-upgrades.log", you should see that at the chosen times, some actions have been carried out. This may just be a check for updates, if nothing new needs to be installed....

Restarting a server?

So, yes, I know, we like to have a big uptime, but...

When someone says to me "my server hasn't rebooted for 3 years", all I can think of is: "OK, but what happens the day you reboot?

The stability of GNU/Linux has been proven, I know very well that it can run for years without rebooting, so as far as I'm concerned, if it reboots from time to time, it doesn't give me any trouble (well, not when I'm using it intensively of course?).

I'd rather know immediately that there's a problem and that a machine won't reboot than to discover it, "forced and obliged", in a moment of "hot" rush.

Conclusion

After Redhat, Ubuntu is moving to 10-year updates of its distribution, which is great news for companies like me who haven't moved over to the cloud and PaaS.

It provides a completely viable alternative in a production environment, while remaining within an Open Source ecosystem.

Linux goes everywhere, a CPU, some RAM and it's running... (well, storage, a network card... too?).

I've got nothing against PaaS, for some customer projects we're with Clever Cloud which runs very well, but I like to keep understanding how my infra runs.

Have a great week and see you next week?