Dan [the] Salmon

During the process of evaluating Proxmox as a Docker host, I found that there are at least 3 methods for running Docker containers in Proxmox. Here are instructions for doing those 3 methods as well as some simple disk speed benchmarks to compare.

The 3 methods I will outline are:

• Docker running in an LXC container
• Docker running in a VM
• Docker running on Proxmox itself

Run Docker in an LXC container

Security warning: This configuration offers very little, if any security to segment the contents of the container from the Proxmox host. This method should not be used in production.

1. On the Proxmox host, edit /etc/modules-load.d/modules.conf to add the aufs and overlay kernel modules

   # /etc/modules: kernel modules to load at boot time.
   #
   # This file contains the names of kernel modules that should be loaded
   # at boot time, one per line. Lines beginning with "#" are ignored.
   aufs
   overlay
   

2. Restart Proxmox host

3. Create an LXC container with your desired settings and OS, making sure to uncheck “unprivileged container”, but don’t start it yet. I’m using Debian 10.

4. In Proxmox, edit the /etc/pve/lxc/<id>.conf file where <id> is the ID given to your container:

   lxc.apparmor.profile: unconfined
   lxc.cgroup.devices.allow: a
   lxc.cap.drop:
   

5. Start the container

6. In the container, create /etc/docker/daemon.json and make the contents:

   {
       "storage-driver": "overlay2"
   }
   

7. Install Docker. The official instructions for Debian can be found here. They boil down to this:

   apt update
   apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
   curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
   add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
   apt-get update
   apt-get install -y docker-ce docker-ce-cli containerd.io
   

Run Docker in a VM

1. Create a VM with any operating system supported by Docker Engine. A list of officially supported distributions can be found here
2. Follow the official instructions for your distribution linked from the previous docs page
3. That’s it!

Run Docker in Proxmox

Note: This method should not be used in a production environment. Like the LXC method, there is very little segmentation between the containers and the Proxmox host. Additionally, the docker daemon runs as the Proxmox root user which is a universally bad idea. This method is the least secure of the 3 listed here.

1. Follow the official documentation for installing Docker Engine on Debian found here

2. You may need to restart Proxmox after installing Docker, but after that it should be good to go

3. Optional: By default, the Docker data-root will be on your local storage where Proxmox itself is installed. If you want Docker to store its data in another location, edit /lib/systemd/system/docker.service and change the ExecStart= line to include the --data-root option. For example, I made a ZFS dataset and pointed Docker to it like this:

   ExecStart=/usr/bin/dockerd --data-root /tank/docker-root -H fd:// --containerd=/run/containerd/containerd.sock

Testing Methodology

I first have to preface these results by saying that this test was very unscientific. This is my first time trying out Proxmox and first time getting “under the hood” of Docker.

• My Docker use case is focused on disk performance. Thus, I did not test CPU performance to see what kind of a virtualization penalty would be introduced with either the LXC or VM method.

• While I tried to keep the test as fair as possible, there were certain variables I couldn’t keep identical across the board, namely: the storage drivers and whether AppArmor was enabled. Here is a breakdown of the variables:

  	CPU Cores	RAM	Storage Driver	Backing Filesystem	AppArmor
  LXC	2	4GB	overlay2	extfs	disabled
  VM	2	4GB	overlay2	extfs	enabled
  Proxmox	2	4GB	zfs	n/a	enabled

• When testing on Proxmox I limited the max CPU cores and memory of the benchmark container using --memory="2g" and --cpuset-cpus="0,1".

• My host consists of an HP Z620 workstation with the following hardware:

  • 1x Xeon E5-2620 CPU
  • 4x 4GB DDR3-1333 RAM modules, 16GB total
  • 4x 1TB WD Blue SATA SSDs configured as a 2x2 ZFS mirror

• Since the VM and LXC container ran on the ZFS pool, I edited the Docker config when testing on Proxmox to move the data-root to a dataset on the ZFS pool. This is why the ZFS storage driver is used.

Disk Speed Benchmarks

Conclusion

Looking at the results, I am not surprised at all that Docker installed directly on Proxmox is the fastest option. What is surprising is that the VM performance was better than the LXC container. I would have thought that since there’s a very thing virtualization layer between the Proxmox host and the LXC container the LXC performance would be better than a fully virtualized VM.

Of course there’s a very good chance that there are some settings I could have changed to get better performance out of all 3 options but I wanted to just get some quick numbers for “out of the box” performance.