Fixing Docker problems in Fedora

In this post, I’ll describe a few problems with Docker in Fedora and how to fix them. In particular, I’ll try to provide an analysis of the problems and the sources of the solutions.

I’ve experienced a few problems with Docker in Fedora (35 and 36). I first installed docker-ce by following the official documentation: https://docs.docker.com/engine/install/fedora/. Things went smoothly at the beginning with the first Docker images, e.g., the “hello-world”, “ubuntu,” and a few others.

However, I started to experience severe problems with this image “mysql:5.7”: the container took a lot of time to start, and I noted that Docker immediately ate my whole RAM, 16 Gb. If I don’t terminate the process, the computer will likely hang in a minute or less.

This is a docker-compose file to recreate the problem:

Save it to “docker-compose.yaml” and start it with “docker-compose up” and see the problem.

You must install the package docker-compose from Fedora’s repositories to verify the problem. I anticipate that the problem is not due to docker-compose because I have precisely the same problems with the above Docker image even if I run it without docker-compose (e.g., during a Maven build).

There’s no problem with a more recent version of the MySQL image, e.g., “mysql:8.0.27”. However, there’s something wrong with Docker in Fedora because the same docker-compose file using “mysql:5.7” does not give me any problem and works perfectly in other Linux distributions like Ubuntu and Arch.

I also experienced slow performance and ram draining when running an ansible molecule test with one of my ansible playbooks, e.g., running a flatpak installation in a Docker container. Again, only in Fedora.

I reported here https://ask.fedoraproject.org/t/docker-very-slow-in-fedora/23214 without getting any solution.

Here are the solutions I found myself.

Use moby engine (and tweak its configuration)

Instead of using “docker-ce”, I tried moby-engine (https://mobyproject.org/), which provides the same “docker” command line tools. The existing docker-compose works with moby as well. As stated here https://fedoramagazine.org/docker-and-fedora-35/, moby-engine is the “Fedora’s way” (though the official Docker package is said to work as well).

Surprise! the above docker-compose file works like magic: it starts fast and does not eat RAM. Also, my ansible molecule tests work perfectly.

Unfortunately, you run into problems using the excellent Testcontainers library. This problem is due to SELinux (that figures!!!). You can use this example project (taken from my TDD book): https://github.com/LorenzoBettini/it-docker-mongo-example. If you have Java and Maven installed, you can clone the repository, enter the directory “com.examples.school,” and run “mvn verify”. When it comes to Testcontainers tests, you get errors of this shape:

If you temporarily disable Selinux:

The Testcontainers tests will succeed.

However, let’s try to fix this problem once and for all.

To understand why moby-engine’s docker command works while docker-ce’s docker does not I inspected the version of the file /usr/lib/systemd/system/docker.service file provided by the moby-engine’s installation and the version provided by docker-ce’s installation.

These are the interesting parts of the docker.service file that comes with the moby-engine package:

While these are the corresponding parts in the version of the file provided by the docker-ce package:

The main differences are:

  • docker-ce uses a simple command to start the Docker daemon
  • moby-engine passes more arguments and relies on additional arguments (the OPTIONS environment variable) read from the additional file /etc/sysconfig/docker, whose contents are listed here:

First, I like this approach of relying on a separate file, which the user can customize. I understand that updates to the moby-engine package will not overwrite this file.

Thus, to avoid the Selinux problem with Testcontainers, it is enough to remove the “–selinux-enabled” argument in the above file. These are the steps:

  • Stop docker, “sudo systemctl stop docker”
  • Modify the file as follows:

  • Restart docker “sudo systemctl daemon reload && sudo systemctl start docker”

Moreover, disabling SELinux in the above file is also required if you, like me, prefer to have the docker directory (by default, it is /var/lib/docker) in another place (e.g., in a standard partition, which is not BTRFS and which is shared among several Linux distributions on the same computer).

Here’s an example of customization of the above file to tell Docker to use a different root directory:

If you use a different root directory with SELinux enabled, you might get errors when running containers of the shape

Fixing docker-ce

By taking inspiration from the configuration of moby-engine, we can also fix docker-ce by tweaking its configuration.

Of course, stop the Docker service (“sudo systemctl stop docker”) before doing the following steps.

It makes no sense to modify the file /usr/lib/systemd/system/docker.service directly because that will be overwritten by docker-ce package updates (I’m sure about that: I’ve seen it).

Instead, we first create the file /etc/sysconfig/docker (which does not exist in docker-ce installation) by removing a few arguments that would not work in docker-ce:

Then, we create another service file to specify the Docker daemon execution command:

And these are the contents of the created file:

Restart docker “sudo systemctl daemon reload && sudo systemctl start docker”. Everything should now work with docker-ce!

Of course, this approach also works with a specified “–data-root” argument, as seen in the moby-engine example.

Conclusions

On a side note, it might be enough to use only a subset of arguments in the docker-ce configuration taken from moby-engine. However, I think I wasted enough time debugging these problems in Fedora, and for the time being, I’m happy with these existing configurations.

I also want to stress my disappointment about how Fedora makes developers’ lives hard in these situations. I decided to debug these Docker problems and find solutions as a challenge. However, my interest in Fedora and my initial appreciation for this distribution significantly decreased. I won’t stop using it shortly, but I will not use it as my daily driver. For that, there’s Arch! 🙂

4 thoughts on “Fixing Docker problems in Fedora

  1. Abraham C

    Hello Lorenzo, I have tried this fix – including moving my data dir to an ext4 partition. However, it did not fix my issue. Do you have any clue what be wrong?

    On my Arch install, the database runs as expected, however on my Fedora install it runs over 4 times as slow.
    Happy to provide debug info.

    Reply
    1. Lorenzo Bettini Post author

      Hi there.
      I ditched Fedora for good a few months ago: it just doesn’t fit my developer needs 🙁
      I use Arch 99% of the time and I don’t regret it 😉

      Thus, I cannot test whether my fix still fixes the problems, I’m afraid.
      All I can say is: are you sure you restarted docker or the computer after applying the fix?
      Are you sure the custom configurations are effectively used?

      Reply
  2. Josue B

    This is valid for Alma Linux 9, I am migrating a very very old app (PHP 4 from Centos 5), and a simple yum update was very very slow, once I added the /etc/sysconfig/docker config and the systemd file, it worked properly. Thanks!

    Reply

Leave a Reply to FabienCancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.