Skip to main content

Installing a RIOS VM

Overview​

The RIOS VM provides a way to install RioRey filtering technology into a virtual machine, rather than using dedicated physical hardware. At a high level, the installation requires the following steps:

  1. Creating the VM (KVM or VMWare)
  2. Configuring the interfaces used by the VM
  3. Configuring and licensing the device in rWeb

In order to start the RIOS VM, you will need to meet the minimum resource requirements to run it in each of the following WAN speeds:

info

CPU cores assigned to a RIOS VM must have SSE4.2 and POPCNT instruction set support (equivalent to Intel 'Nehalem'/microarchitecture level 'x86-64-v2' or later).

For VMs with RIOS versions > 8.4.0, these resource minimums are used:

ThroughputMemoryCPU CoresStorage
1 Gbps13 GiB4 Cores4 GB Virtual Storage
5 Gbps18 GiB5 Cores4 GB Virtual Storage
10 Gbps24 GiB8 Cores4 GB Virtual Storage
40 Gbps82 GiB25 Cores4 GB Virtual Storage
100 Gbps120 GiB30 Cores4 GB Virtual Storage

Versions prior to 8.4.0 use these minimums:

ThroughputMemoryCPU CoresStorage
1 Gbps13 GiB3 Cores4 GB Virtual Storage
5 Gbps18 GiB4 Cores4 GB Virtual Storage
10 Gbps24 GiB6 Cores4 GB Virtual Storage
100 Gbps (requires RIOS 8.2.0 or higher)80 GiB28 Cores4 GB Virtual Storage

It is recommended that you read the VM Performance Considerations section in addition to the interface discussion below. If deploying a 100G VM, please also read the 100G RIOS VM Considerations section as well.

Interface Considerations​

Once the basic configuration is complete, you will next need to configure the network interfaces of the device. Depending on your VM deployment, your networking architecture, and the available resources on the host machine, the configuration could be drastically different. All RIOS VMs, however, will need three things:

  1. A management interface
  2. A WAN interface, on which ingress traffic will be filtered
  3. A LAN interface

Once configured, the VM should be deployed within your architecture so that it is able to connect to rWeb via the management interface, the traffic you want filtered is coming in on the WAN interface, and the resulting traffic will be on the LAN interface. Filtering is always done in the WAN to LAN direction. No filtering is done in the LAN to WAN direction. This also means you can return traffic over a different network path, which may be beneficial in cases where you don't have additional cores for separate LAN processing.

For the management interface, RIOS is expected to use a static IP. This IP must be reachable from where you have installed rWeb. When you configure the device in rWeb, rWeb will initiate the connection to begin management of the RIOS VM.

The biggest performance considerations for filtering revolve around the configuration of the WAN interface. There are currently two supported methods of configuration, listed below. A brief discussion of each method will follow:

  • Virtual Interface
  • PCI Passthrough
danger

For all configuration types, use either virtio or e1000 as the model for the interface. rtl8139 is explicitly not supported and the VM will not function as expected.

Virtual Interface​

Virtual interfaces are the most straightforward to configure, but have the worst performance characteristics. This is due to the high overhead imposed by host-to-guest I/O operations. Virtual interfaces provide acceptable performance for the management interface, but for better performance on the data plane it is strongly suggested you configure either PCI Passthrough or SR-IOV.

PCI Passthrough​

PCI Passthrough gets good performance, but requires dedicating a host machine network card for use by the RIOS VM. The RIOS VM uses the Data Plane Development Kit (DPDK) for data plane operations, so the card selected must be on the formally supported list of hardware shown here. Our current internal testing uses the Intel 82599 and XL710 cards, so we recommend sticking with Intel NICs if possible. For the 100G platform we are using the Mellanox ConnectX-5 Ex MCX516A-CDA_Ax_Bx.

Unsupported Configuration​

SR-IOV​

While currently unsupported, Single-root Input/Output Virtualization (SR-IOV) can also be configured. SR-IOV is similar to PCI passthrough, but allows the host network card to be shared among multiple VMs. It has good performance characteristics at the cost of increased complexity of configuration. When using SR-IOV, a promiscuous Virtual Function (VF) must be injected into the VM in order for the interface to function properly. A full discussion of how to configure SR-IOV is out of scope for this guide as it is unsupported, but a more detailed description of configuring SR-IOV in the KVM environment can be found here.

VM Performance Considerations​

General​

It is highly recommended that you use Virtio for the management plane of VM devices. VM devices will get the most performance on the data plane by having a network card configured though PCI Passthrough. Additionally, for systems using Non-Uniform Memory Access (NUMA), better performance is achieved by having your VM be scheduled on the same NUMA node that is handling your network interfaces. It is also important that the NUMA node have enough local memory available to meet the minimum requirements of your selected VM.

Where possible, RIOS VMs should have dedicated resources isolated from other virtual machines that may be running on the same host.

We do not recommend using Simultaneous Multi Threading (SMT) technologies (e.g. Intel Hyper-Threading), as after testing and evaluation it results in a peak performance decrease and increases per packet latency variation.

VMware ESXi​

Every time a VM is powered on, it is randomly assigned a desired NUMA node, which may not line up with your physical hardware. For more reliable performance, set the number of cores allocated to the RIOS VM to the recommended number for your VM size (1G, 5G, 10G, or 100G), and then use the advanced numa.nodeAffinity controls to pin the VM to the same NUMA node as the physical NIC. If you have a NUMA domain that is smaller than the VM size, you'll probably need to use individual CPU pinning - priority for the same domain as the physical would be first the WAN cores, then the LAN core, then the OS core.Your motherboard manufacturer should have information available about NUMA layout of PCI cards and CPU cores. There are also commands in ESXi to help determine this information:

  • esxcli hardware memory get: Use this command to confirm the amount of memory available per NUMA node and that this meets the requirements for your VM.
  • esxcli hardware cpu list: Use this command to identify the node each CPU core is associated with.

If you have a running VM, the vmdumper command can give the current NUMA status of a VM to confirm where cores are being scheduled.Setting the Latency Sensitivity to 'high' will turn on a whole set of optimizations and tuning parameters that should make performance better and more consistent. The tradeoff is that more resources will be reserved for this VM. See more information here.

100G RIOS VM Considerations​

Reference Platform​

The 100G VM was primarily tested with the following supported components and configuration (using PCI passthrough):

  • CPU - AMD EPYC 7502P 32-Core
  • Motherboard - PCI Gen 3 Motherboard, single socket
  • Memory - 256 GB, 8x32GB DDR4
  • Network Interface Card (NIC) - Mellanox ConnectX-5 Ex MCX516A-CDA_Ax_Bx

    • The tested and supported version of firmware for the card has changed alongside the following RIOS releases:

      RIOS ReleaseFirmware Version
      9.0.016.35.8008
      8.5.016.35.3502
      8.1.016.29.2002

    • All firmware versions are downloadable here. Installing or upgrading firmware will depend on your VM architecture. It is recommended to install Mellanox Firmware Tools (MFT) for your environment and use it for installation as described in the link. There is also an open source version of this toolset, available here, that we also support.

    • Two additional firmware configuration changes are recommended (using mlxconfig from MFT, or mstconfig from the open source toolset):

      • PCI_WR_ORDERING=1
      • CQE_COMPRESSION=1

    • Host OS Driver updates are not required if using PCI passthrough as recommended. If doing a vSwitch style deployment, driver updates will be required. Both ESXi and general Linux drivers are available.

NUMA Configuration​

Minimally, in a multi-socket environment, we recommend using CPU affinity to limit the VM to the host CPU socket that is directly connected to the network card as described in our general performance recommendations. The selected socket should also have enough local memory available for the VM. For our reference platform specifically, managing NUMA within a single socket has not been necessary, but it is ideal to have WAN and LAN cores pinned 1-to-1 with isolated host cores. Additional resources on NUMA management for libvirt and ESXi are available in their respective links.

Suboptimal Configurations​

A 100G VM RIOS requires many more cores than the 10G or lower VM RIOS devices. In certain cases, your host may not have enough cores on a single socket to be configured as described above. You may need to make configuration changes depending on your specific host resource availability. Below, in no particular order, are a variety of configuration changes you can make that will allow you to get a 100G RIOS VM operational with some degradation in performance.

  • Double up WAN/LAN cores- Normally we recommend pinning WAN and LAN to isolated cores. If short of cores, you can pin LAN cores to the same cores as the WAN processes. Computationally WAN cores are doing considerably more work than LAN cores, so this does increase the workload but by a relatively sustainable amount.
info

In RIOS 8.4.0 the layout manager will recommend doubling across all WAN/LAN cores due to the increase in WAN core usage and current limitations on total number of cores. This will currently produce the best performance for a 100G VM.

  • Move System/LAN cores to a different node- Similarly, you can instead move the System and LAN cores to a different NUMA node. Here the performance penalty is largely incurred by LAN side traffic being processed away from the network card.
  • Allocate fewer cores- You can also allocate fewer cores than the declared minimum. While it will always be able to use the underlying 100G interface card, actual VM throughput will drop based on how many cores below the minimum you are using. There is no direct formula, but expect throughput scale roughly linearly between the declared minimum of 28 cores for 100G and 6 cores for 10G. It is important to note that while you can allocate fewer cores, you must always meet the minimum memory requirements of the 100G VM... if less is allocated the VM will not run. This configuration is useful for situations where you may be using 100G physically but have only provisioned for a lower level of bandwidth over it.
  • Use Simultaneous Multi Threading (SMT)- Though we don't recommend this in general, enabling something like Intel Hyper-Threading can give you more logical cores to work with. In this case you are trading reduced overall throughput by allocating fewer cores (with SMT disabled) for a peak performance hit and increased packet latency variation (with SMT enabled).

vSwitch Limitations​

Be aware that when using vSwitch instead of PCI passthrough the VM host can encounter issues when running traffic at high packet rates. For this reason we recommend using PCI passthrough for 100G devices more strongly then lower bandwidth VM RIOS devices.