Vertica Test Results for MDS Security Patches

Several customers have asked us how Intel’s recent MDS vulnerability patches will impact the performance of their Vertica databases. Vertica is known for its performance. So, anything that may impact its speed is a major concern for both us and our customers.

Vertica engineers have finished a round of testing the OS and microcode fixes for this most recent set of security vulnerabilities found in Intel processors. This blog post explains how we tested, and what we found.

Background
In May, Intel and third-party security researchers announced a new set of security vulnerabilities in Intel processors. These vulnerabilities are in addition to the Meltdown and Spectre security vulnerabilities that Intel and researchers disclosed in 2018. These earlier issues are covered in our previous blog post UPDATE: Vertica Test Results with Microcode Patches for the Meltdown and Spectre Security Flaws.

These new security issues are collectively known as the Microarchitecture Data Sampling (MDS) vulnerabilities. They rely on exploiting the internal architecture of Intel microprocessors that developers usually do not use directly. In fact, most likely do not even know they exist. Like the earlier Meltdown and Spectre vulnerabilities, these issues use loopholes in the processor’s speculative execution features. These features make the processor more efficient by having it start processing new instructions before it has completed processing earlier instructions. In theory, if the earlier instructions result in an error (for example, trying to access memory that is off-limits) the processor can discard the speculatively executed instructions without any side effects. As it turns out, cleverly written code can deduce the data that was used by the speculatively executed instructions, allowing malicious code the ability to peek at data it should not be allowed to access.

The individual vulnerabilities are:

• Microarchitectural Fill Buffer Data Sampling (MFBDS) CVE-2018-12130, also known as ZombieLoad. This vulnerability uses speculative execution to determine the contents of the buffers that the processor uses when loading data from memory.
• Microarchitectural Store Buffer Data Sampling (MSBDS) CVE-2018-12126 also known as Fallout. This vulnerability uses speculative execution to determine the contents of the processor’s internal store buffer. This buffer holds data the processor is writing to cache or system memory.
• Microarchitectural Load Port Data Sampling (MLPDS) CVE-2018-12127 and CVE-2019-11091 also known as Rogue In-flight Data Load (RIDL). This is a set of vulnerabilities that read data from several different internal microprocessor structures, such as the load ports. The processor uses these ports to read data from memory or I/O operations.

In a nutshell, these vulnerabilities allow malicious code to read data from other processes running on the system. This data includes both user-level and privileged operating system information. The vulnerabilities are a concern on any system that runs code supplied by third parties who cannot be absolutely trusted. One example exploit of an MDS vulnerability was written in JavaScript, meaning that systems that simply load a web page via a browser are at risk.

These issues affect most of the Intel processors produced since 2011. Intel has published a full list of the affected processors in the article Deep Dive: CPUID Enumeration and Architectural MSRs. They have stated that their most recent processors (those produced in 2019) are not susceptible to these vulnerabilities.

The Industry Response to MDS
Intel has issued a set of microcode updates for the processors affected by the MDS vulnerabilities. Microcode is a low-level system code that affects how a processor operates. These updates are loaded into the processor by the operating system.

The Linux kernel developers have incorporated Intel’s microcode changes into the kernel. They have also implemented the required OS-level changes to help mitigate the issues. Linux distribution vendors such as Red Hat and Canonical have released updated kernels for their currently-supported OS releases.

Virtual machine and cloud providers such as VMware, Amazon’s AWS, and Microsoft’s Azure have also patched their products to mitigate these vulnerabilities. Cloud environments are especially vulnerable to these issues, as your virtual system shares a processor with unknown third parties running software that you do not control.

These steps were coordinated with Intel and the security researchers, so the patches were available when Intel and researches announced the vulnerabilities.

Potential Performance Impacts
Your processor will get Intel’s microkernel changes when you upgrade your Linux kernel. These changes require the processor to take additional steps to clear data from its internal data structures. Therefore, they do have an impact on system performance. See the Phoronix blog post The Performance Impact Of MDS / Zombieload Plus The Overall Cost Now Of Spectre/Meltdown/L1TF/MDS for some measurements of this impact. This performance impact is unavoidable. You will eventually get the updated microcode (if you haven’t already). The only way to avoid this impact is to refuse all kernel upgrades, which is impractical and insecure.

The Linux kernel-level changes can be tweaked using several kernel settings to enable or disable certain behaviors. These are the settings that we have focused on when testing the performance impact of the kernel updates.

Some security researches state that to fully mitigate these vulnerabilities, you must disable the processor’s Hyper Threading (HT) feature. Red Hat refers to this feature as Simultaneous Multi-Threading (SMT). It lets two software threads run at the same time on a single processor core. That doubles the number of processor cores available to the operating system. In practice, HT can boost system performance by up to 30-40%. Most of the recently disclosed vulnerabilities exploit weaknesses in HT. That’s why some experts believe disabling HT is the only way to completely mitigate these vulnerabilities.

Opinions are mixed on whether MDS vulnerabilities warrant disabling HT. Red Hat recommends that customers disable it. Intel disputes that HT needs to be disabled in many cases. Ubuntu recommends “disabling Hyper-Threads on affected systems if the system is used to execute untrusted or potentially malicious code.” This implies that you should disable HT on any platform where you do not control all processes running on the host machine.

Ultimately, you need to weigh the performance impact of disabling HT against the security threat posed by MDS exploits in your own environment. To assess the impact of disabling HT, we tested disabling HT while using the recommended kernel settings.

Kernel Settings Overview
The MDS mitigations are controlled by the same kernel settings that are used for the Meltdown and Spectre mitigations. These settings are:

• PTI (Page Table Isolation) protects against Meltdown. Implemented in the kernel.
• IBRS (Indirect Branch Restricted Speculation) protects against Spectre Variant 2. Implemented in the kernel and CPU microcode.
• IBPB (Indirect Branch Prediction Barrier) protects against Spectre Variant 2. Implemented in the kernel and CPU microcode.
• RETP (Retpoline) is a mitigation strategy developed by the Google Project Zero team that worked on the Spectre attacks. The idea is to avoid using indirect JMP or CALL instructions that would be vulnerable to Spectre.

Some of these settings are restricted by the operating system. For example, RETP and IBPB are read-only in Red Hat 6.10. Consult your Linux distributions documentation for details on configuring these kernel settings.

As discussed in our previous blog post, the recommended values for these settings depend on the processor family:

Setting	Pre-Skylake Processors	Skylake Processors
PTI	1	1
IBRS	0	1
RETP	1	0
IBPB	1	1

How We Tested the Patches
As mentioned earlier, the microcode changes provided by Intel are part of the newer Linux kernels, and are basically unavoidable. Therefore, all our testing was done on kernels containing the updated microcode and associated kernel changes. The focus of our testing was determining the performance impact of kernel settings that control the mitigation features. The baseline for the tests were disabling as many of the mitigation settings of the earlier Meltdown and Spectre vulnerabilities as possible.

Vertica engineers carried out these tests using Vertica version 9.2.1 running on Red Hat versions 6.10 and 7.6. As with the previous rounds of testing, we used a select set of queries from the industry-standard TPC-H benchmark. Specifically, we ran queries qtpch1 through qtpch22. Each query was run three times, and the results averaged together.

The database we used in testing had:

• 125 GB of data
• A catalog size of 30 MB
• 8 tables
• 14 projections
• 104 storage containers
• 896 ROS catalog objects
• 338 ROS files

For hardware, we used 4-node clusters running on HP DL360 Gen9 servers configured with two Xeon E5-2698 v3 processors. These processors are members of the Haswell processor family which are among those that are vulnerable to MDS.

The Red Hat version 6.10 cluster had kernel version 2.6.32-754.15.3.el6.x86_64 with microcode version microcode_ctl-1.17-33.13.el6_10.x86_64. We tested the following combinations of kernel and Hyper Threading settings:

PTI Enabled	IBRS Enabled	RETP Enabled	IBPB Enabled	Hyper Threading
0	0	1	1	Enabled
1	1	1	1	Enabled
1	0	1	1	Enabled
1	0	1	1	Disabled

Note: both RETP and IBPB are read-only settings in Red Hat 6.10.

The Red Hat 7.6 cluster had kernel version 3.10.0-957.21.3.el7.x86_64 with microcode version microcode_ctl-2.1-47.4.el7_6.x86_64. We tested the following combinations:

PTI Enabled	IBRS Enabled	RETP Enabled	IBPB Enabled	Hyper Threading
0	0	0	1	Enabled
1	1	0	1	Enabled
1	0	0	1	Enabled
1	0	1	1	Enabled
1	0	1	1	Disabled

Note: IBPB is read-only in Red Hat 7.6.

Test Results
The following chart shows the results on Red Hat 6.10 for each of the 22 TPC-H queries we ran with each set of kernel settings.

As you can see, the results for most of the settings are close, except for PTI=1, IBRS=1, RETP=1, and IBPB=1 with HT on. This combination shows a significant slowdown. The following chart shows the percent increase or decrease in speed for each set of settings to the baseline settings of PTI=0, IBRS=0, RETP=1, and IBPB=1 with HT on.

A negative value in this chart means the settings were faster than the baseline setting. You can see that the setting PTI=1, IBRS=0, RETP=1, IBPS=1 (the recommended settings) with HT off has a slight (1.1% faster) advantage over the baseline settings. However, this result in within the error of margin for the test. It does show, however, that disabling HT did not have a major impact on the benchmark test.

The following chart shows the results for Red Hat 7.6 for each query.

Here, the results are more closely grouped. The summary in the following chart shows the averages across all of the queries.

You can see that the settings PTI=1, IBRS=1, RETP=0 and IBPB=1 with HT on has the worst results (4.4% slower) than the baseline settings. The other setting combinations are slightly faster than the baseline in these results. All these results are probably within the margin of error for the test. The recommended settings for this processor family (PTI=1, IBRS=0, RETP=1, IBPS=1) is just a fraction of a percent slower than the fastest results. Disabling HT when using the recommended settings has a negligible impact on the results.

Conclusions
Our testing has shown that for Red Hat 6.10 and 7.6 with an MDS-vulnerability-patched kernel, the recommended settings of PTI=1, IBRS=0, RETP=1, IBPS=1 for Haswell family processors had performance similar to the results of disabling the vulnerability mitigations.

The results also show that disabling the processor’s HT feature did not significantly impact on the Vertica database’s performance. The test results with HT on and off were within one percent of each other, which is within the margin of error for these tests. These results imply that your database performance will not suffer if you choose to disable HT to mitigate the MDS vulnerabilities.

Note that these benchmark results are based on testing a single processor family (specifically, the Haswell family). Other processor families may have different results. For example, in our testing of the Meltdown and Spectre mitigations, we found significant differences between the older Haswell architecture and the newer Skylake processor families.