Since the company's inception in 2011, we have used a variety of virtualization technologies. These technologies' benefits are rather indisputable: cost reduction and ease of management (backup, migration, cloning, etc.) In fact, virtualization has become a must in the field of information technology.
Although we have good amount of experience with virtualization technologies such as VirtualBox and VMWare, our use of KVM and LXC is fairly recent. For some time now, we have migrated our entire infrastructure to Proxmox, an all-in-one virtualization solution.
To get the most out of this new infrastructure, we want to establish the best configurations to get the maximum performance possible from our Virtual Machines.
To begin our analysis, we want to compare the performance between the two virtualization technologies available with Proxmox, KVM and LXC:
- KVM Kernel-based Virtual Machine) is a hypervisor built into the Linux core. KVM is a virtualization technology similar to what is offered by VirtualBox or VMWare. The hypervisor emulates all the material. Thus, the operating system believes it is running on a physical machine. It is, therefore, possible to install an operating system without alteration.
- LXC (Linux Containers) is another virtualization technology that is also integrated into the Linux core. However, it uses isolation to separate the performance of each machine. Compared to KVM, the hardware is not emulated; the host system shares it. Thus, each Virtual Machine uses the same core. It would therefore be impossible to install MS Windows in an LXC container. LXC is a virtualization technology similar to what is offered by OpenVZ and Docker.
Given the nature of LXC, it is expected to outperform KVM. This is what we want to confirm with our tests.
To execute all our performance tests reliably and reproducibly, we used Phoronix Test Suite.
Here are the technical specifications of the physical server and the two Virtual Machines.
| Physical host | LXC | KVM | |
|---|---|---|---|
| CPU | 2 x Intel Xeon E5620 @ 2.40GHz (16 Cores) | 2 Cores | 2 Cores |
| Motherboard | Supermicro X8DTL | Supermicro X8DTL | QEMU Standard PC (i440FX + PIIX 1996) |
| Chipset | Intel 5500 I/O + ICH10R | N/A | Intel 440FX- 82441FX PMC |
| Memory | 24576MB | 512MB | 512MB |
| Disks | 4 x 250GB Seagate ST250DM000-1BD14 | 8GB subvol (recordsize=128K) | 8GB zvol (volblocksize=8K) |
| Network card | Intel 82574L Gigabit Connection | N/A | Red Hat Virtio device |
| Operating system | Debian GNU/Linux 8 (Proxmox VE) | Debian 8.7 | Debian 8.7 |
| Kernel | 4.4.35-2-pve (x86_64) | 4.4.35-2-pve (x86_64) | 3.16.0-4-amd64 (x86_64) |
| Compiler | GCC 4.9.2 | GCC 4.9.2 | GCC 4.9.2 |
| File system | zfs | ext4 | ext4 |
Results
Multiple Sequence Alignment
In this test, which verifies the CPU's power by performing DNA alignment calculations (no multi-thread), we see that KVM has a considerable lead over LXC. The big difference between the physical host and KVM is explained by the number of CPUs allocated: 16 CPU for the host vs. 2 CPU for KVM.
CAMELLIA256-ECB Cipher
This test also allows you to check the performance of a single CPU by performing cryptography calculations (no multi-thread). All three tests yield similar results. We note that LXC has a slight lead over KVM.
AI Chess Performance
This test verifies the performance of a single CPU by simulating a game of chess (no multi-thread). All three tests yield similar results. We note that LXC has a slight lead over KVM.
Poisson Pressure Solver
This test also verifies the performance of a single CPU by solving linear equations (no multi-thread). All three tests yield similar results. We note that LXC has a slight lead over KVM.
7zip Compress Speed Test
This test essentially verifies the performance of the CPU by compressing data (with multi-thread). The physical host has a large lead due to the number of CPUs allocated: 16 CPU for the host vs. 2 CPU for KVM. We note that LXC has a slight lead over KVM.
C-Ray Total Time
This test verifies the performance of the CPU by calculating ray tracing. This test allows you to check the CPU's performance in calculating floating numbers (with multi-thread). The physical host has a large lead due to the number of CPUs allocated: 16 CPU for the host vs. 2 CPU for KVM. We note that LXC has a slight lead over KVM.
WAV To FLAC
This test verifies the performance of a single CPU by converting a music file from THE WAV format to FLAC (no multi-thread). All three tests yield similar results. We note that LXC has a slight lead over KVM.
WAV To MP3
This test verifies the performance of a single CPU by converting a music file from THE WAV format to MP3 (no multi-thread). All three tests yield similar results. We note that LXC has a slight lead over KVM.
Apache Static Web Page Serving
This test represents a typical server use that tests the CPU, discs and memory. The physical host obviously provides better performance, followed by LXC and KVM.
The results of this test confirm our initial hypothesis: LXC performs better than KVM.
Aio Stress Random Write
This test can be akin to using a database that does a lot of writing to disk. Surprisingly, KVM has a slight lead over LXC.
iozone: Size: 4GB - Disk Test: Read Performance
This test verifies random disk playback. The results provided by this test are not representative. The discs in the server are not able to provide 1300MB/s. It must be inferred that the system cache greatly increases the results.
We must conclude the following: LXC can get the most performance from the system cache, while KVM cannot take advantage of it probably because it doesn't have as much memory as the host.
iozone: 4GB - Disk Test: Write Performance
This test verifies random writing to disk. Here, too, we must understand that we measure the cache of data. The results provided by this test are not representative.
Pfam Database Search
This test simulates a query in a database. Unsurprisingly, KVM is still the last in line, but offers reasonable performance.
LZMA 256MB File Compression
This test essentially verifies the performance of a single CPU by compressing data with the LZMA algorithm. ~~I can't explain KVM's lead in this test.~~ (edit) LXC still has a small lead over KVM.
PHP Benchmark Suite
This test essentially verifies the performance of a single CPU by running PHP scripts. All three tests yield very similar results. LXC still has a small lead over KVM.
Illumination Renderer 100 Samples
This test verifies the performance of the CPU by calculating light trajectories to generate an image. The physical host has a large lead due to the number of CPUs allocated: 16 CPU for the host vs. 2 CPU for KVM. We note that LXC has a slight lead over KVM.
Results Tables
| Hôte physique | LXM | KVM | |
|---|---|---|---|
| mafft: Multiple Sequence Alignment (seconds) | 6.75 | 25.14 | 20.71 |
| gcrypt: CAMELLIA256-ECB Cipher (microseconds) | 3473 | 3493 | 3540 |
| tscp: AI Chess Performance (nodes/sec) | 545872 | 544252 | 559410 |
| himeno: Poisson Pressure Solver (MFLOPS) | 1109.26 | 1111.99 | 1054.58 |
| compress-7zip: Compress Speed Test (MIPS) | 23825 | 5034 | 4271 |
| c-ray: Total Time (sec) | 19.48 | 76.85 | 79.2 |
| compress-pbzip2: 256MB File Compression (sec) | 7.94 | 208.31 | |
| encode-flac: WAV To FLAC (sec) | 13.22 | 13.14 | 14.7 |
| encode-mp3: WAV To MP3 (sec) | 22.22 | 22.18 | 22.39 |
| apache: Static Web Page Serving (request/sec) | 15595.2 | 7008.57 | 5381.83 |
| aio-stress: Random Write (MB/s) | 646.06 | 309.01 | 314.95 |
| iozone: Size: 4GB - Disk Test: Read Performance (MB/s) | 1370.47 | 1319.17 | 375.62 |
| iozone: Size: 4GB - Disk Test: Write Performance (MB/s) | 334.74 | 351.66 | 257.11 |
| hmmer: Pfam Database Search (sec) | 13.63 | 38.99 | 43.43 |
| compress-lzma: 256MB File Compression (sec) | 456.85 | 455.46 | 524.6 |
| phpbench: PHP Benchmark Suite (score) | 101832 | 103100 | 101481 |
| smallpt: Global Illumination Renderer; 100 Samples (sec) | 80 | 409 | 418 |
Analysis of results
With these results, it is observed that CPU-related tests provide comparable results in LXC and KVM if the number of CPUs allocated to Virtual Machines is the same. It is also observed that the performance of the physical host, LXC and KVM are all very similar when the test verifies the performance of a single CPU. It should be noted, however, that LXC wins with a small lead in the majority of tests.
In terms of disk-related tests, there is a significant difference between LXC and KVM. KVM being slower than LXC in all writing or disc playback tests.
This suggests that KVM and LXC provide similar performance if your application is strictly CPU-related. LXC and KVM make the most of the CPUs allocated to them. There is almost no loss of performance compared to the physical host.
On the other hand, when writing to disk or disk playback is involved, KVM and LXC don't perform as well as the physical host. Reading and writing to the disc is, therefore, the Achilles heel of virtualization. We need to move in this direction if we want to improve the performance of our Virtual Machines.
Overall, LXC clearly performs better than KVM. That said, KVM is not far behind.
Based on these results, should you convert your entire environment to LXC? Perhaps not. The effort required to convert multiple Virtual Machines compared to the small performance gain probably doesn't warrant it. Besides, performance is just one of several factors in your decision-making between KVM and LXC. In addition, in future articles, we will determine the best configuration parameters to improve KVM and LXC disk writing and playback performance.