Ensuring Performance for Real-time Media Packet Processing in OpenStack – Part 2

Overcommit settings. Override OpenStack’s default overcommit settings, and ensure a 1:1 ratio for CPU and RAM.

NUMA placement. Although each NUMA node will have dedicated bus to memory that is local to the node, it is also possible to access remote RAM across a bus that is shared with all nodes. When accessing remote RAM on shared bus or using a NIC that is not local to the host, is it is possible to run into problems such as: unintended cache synchronization among the NUMA nodes; I/O performance degradation; or a waste of resources in the shared intra-node bus. To avoid these drawbacks, place guests such that they are entirely confined within a single non-uniform memory access (NUMA) node and ensure spawning of instances with I/O device locality awareness.

vCPU topology. By default, in OpenStack, the virtual CPUs in a guest are allocated as standalone processors. However, this is not necessarily valuable for all applications. For our specific application, hypervisor overhead can be reduced if the host topology is also configured in the guest.

CPU pinning. From the hypervisor's perspective, the virtual machine appears as a single process which needs to be scheduled on the available CPUs. While the NUMA configuration above means that memory access from the processor will tend to be local, it is possible for the hypervisor to then choose to place the next scheduled clock tick on a different processor. Unfortunately, this can lead to less memory locality and increases the possibility of cache misses, thus a guest VM might starve for scheduling even if the VM is ready to get scheduled.

Therefore, our recommendation is to have the guest vCPUs pinned to the host CPUs and dedicated to it, so other VMs on the same host cannot use them. Compute resources should host either only pinned VMs or unpinned VMS, but not a mix of the two. And if hyper-threading is enabled on the host retain the default thread policy.

Specifying the use of large pages. Use the Linux kernel feature "transparent huge pages" (THP) and to explicitly allocate huge pages to application RAM allocations when practical to do so. The allocation of huge pages reduces the requirement of page allocation at runtime being dependent on the memory usage, reduces the hypervisor overhead, and VMs can get the RAM allocation that will boost their performances.

Isolation of CPUs for host processes. In order to have all the pinned CPU dedicated for the guest vCPU and no interferences with the host level processes, it is always better to isolate the CPUs to be used by the VMs.

Turn-off / non-usage of settings. There are several OpenStack settings that need to be turned-off or otherwise not used: Get better memory access and ensure low latency in memory access, by NOT using Swap memory on the compute host.

• When using the large page option, a RAM over-commit ratio is no longer used, because there must be a 1-1 mapping between guest RAM and host RAM, and the host OS won't consider any large pages allocated to the guest for swapping. In conjunction, we believe it makes sense to turn off over-commit for vCPUs too.
• Kernel shared memory (KSM) can provide significant improvements in the utilization of RAM when many identical guest OS are run on the same host, or guests otherwise have identical memory page contents. However, the cost of KSM is increased CPU usage and this overhead may cause the applications to run more slowly, so we recommend turning off KSM.
• CPU model setting for the VM. One option in the Nova configuration allows the model of CPU visible in the guest to be configured between different alternatives. For our compute-intensive application, we recommend the host BIOS power setting to be configured for maximum performance and the CPU mode to be set to “host-pass through” or “host-model”, so that all the CPU flags are exposed from the host to the guest.