Hyper-V: Receive Side Scaling (RSS-Settings (VMQ)): Unterschied zwischen den Versionen
Admin (Diskussion | Beiträge) |
Admin (Diskussion | Beiträge) K |
||
| (3 dazwischenliegende Versionen des gleichen Benutzers werden nicht angezeigt) | |||
| Zeile 3: | Zeile 3: | ||
''Note Because hyper-threaded CPUs on the same core processor share the same execution engine, the effect is not the same as having multiple core processors. For this reason, RSS does not use hyper-threaded processors. | ''Note Because hyper-threaded CPUs on the same core processor share the same execution engine, the effect is not the same as having multiple core processors. For this reason, RSS does not use hyper-threaded processors. | ||
| − | ''' | + | '''A physical host can have Hyper-V enabled and still use RSS.''' VMQ is only enabled on a NIC when there is a vSwitch attached to it. If you have two NICs in a server it is perfectly acceptable to have the NICs unteamed, one attached to a vSwitch and the other used for host networking traffic. When using this configuration, one NIC will use VMQ and the other will use RSS. |
VMQ leverages the same hardware queues on the NIC as RSS, the ones that interrupt different cores. However with VMQ the filters and logic associated with queue distribution of the packets is different. Rather than one physical device utilizing all the queues for its networking traffic, '''these queues are balanced among the host and VMs on the system.''''' | VMQ leverages the same hardware queues on the NIC as RSS, the ones that interrupt different cores. However with VMQ the filters and logic associated with queue distribution of the packets is different. Rather than one physical device utilizing all the queues for its networking traffic, '''these queues are balanced among the host and VMs on the system.''''' | ||
| Zeile 9: | Zeile 9: | ||
== Important / to consider == | == Important / to consider == | ||
| − | *Do not use Core 0! | + | *Do not use Core 0! (Because of the Parent Partition (Management OS),...) |
*Do not use Hyperthreaded Cores! | *Do not use Hyperthreaded Cores! | ||
*NUMA-Distance should be 0! | *NUMA-Distance should be 0! | ||
*To fully utilize the CPUs, the '''number of RSS Receive Queues should be equal to or greater than Max Processors!''' | *To fully utilize the CPUs, the '''number of RSS Receive Queues should be equal to or greater than Max Processors!''' | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| Zeile 58: | Zeile 50: | ||
Set-NetAdapterVmq -Name <NIC-Name> -BaseProcessorNumber <erster Core, nicht Core 0> -MaxProcessors <anzahl Cores> -MaxProcessorNumber <letzer Core> -NumaNode <NUMA-Node> | Set-NetAdapterVmq -Name <NIC-Name> -BaseProcessorNumber <erster Core, nicht Core 0> -MaxProcessors <anzahl Cores> -MaxProcessorNumber <letzer Core> -NumaNode <NUMA-Node> | ||
</source> | </source> | ||
| − | Die Anzahl Queues können ebenfalls mit dem Parameter ''-NumberOfReceiveQueues'' gesetzt werden. Hier wird empfohlen die Einstellungen auf | + | Die Anzahl Queues können ebenfalls mit dem Parameter ''-NumberOfReceiveQueues'' gesetzt werden. Hier wird empfohlen die Einstellungen auf Automatisch zu belassen.<br> |
Wenn genug Cores pro CPU vorhanden sind wird für eine 1Gig NIC -> 8 Queues empfohlen und für eine 10Gig NIC und höher -> 16 Queues. | Wenn genug Cores pro CPU vorhanden sind wird für eine 1Gig NIC -> 8 Queues empfohlen und für eine 10Gig NIC und höher -> 16 Queues. | ||
| Zeile 69: | Zeile 61: | ||
| − | Die Optimale Verteilung der Queues | + | Die Optimale Verteilung der Queues wäre folgende:<br> |
''Blau markiert die physischen Cores.'' <br> | ''Blau markiert die physischen Cores.'' <br> | ||
<span style="color:red">'''Es dürfen keine Hyperthreadete Cores verwendet werden!'''</span><br> | <span style="color:red">'''Es dürfen keine Hyperthreadete Cores verwendet werden!'''</span><br> | ||
| Zeile 96: | Zeile 88: | ||
| − | Die Optimale Verteilung der Queues | + | Die Optimale Verteilung der Queues wäre folgende:<br> |
''Blau markiert die physischen Cores.'' <br> | ''Blau markiert die physischen Cores.'' <br> | ||
<span style="color:red">'''Es dürfen keine Hyperthreadete Cores verwendet werden!'''</span><br> | <span style="color:red">'''Es dürfen keine Hyperthreadete Cores verwendet werden!'''</span><br> | ||
| Zeile 115: | Zeile 107: | ||
| − | + | More Informations: <br> | |
| − | + | ||
'''RSS:''' https://docs.microsoft.com/en-us/windows-hardware/drivers/network/introduction-to-receive-side-scaling <br> | '''RSS:''' https://docs.microsoft.com/en-us/windows-hardware/drivers/network/introduction-to-receive-side-scaling <br> | ||
'''VMQ Best Practise:''' https://blogs.technet.microsoft.com/networking/2016/01/04/virtual-machine-queue-vmq-cpu-assignment-tips-and-tricks/ <br> | '''VMQ Best Practise:''' https://blogs.technet.microsoft.com/networking/2016/01/04/virtual-machine-queue-vmq-cpu-assignment-tips-and-tricks/ <br> | ||
Aktuelle Version vom 25. September 2018, 13:03 Uhr
Receive side scaling (RSS) is a network driver technology that enables the efficient distribution of network receive processing across multiple CPUs in multiprocessor systems.
Note Because hyper-threaded CPUs on the same core processor share the same execution engine, the effect is not the same as having multiple core processors. For this reason, RSS does not use hyper-threaded processors.
A physical host can have Hyper-V enabled and still use RSS. VMQ is only enabled on a NIC when there is a vSwitch attached to it. If you have two NICs in a server it is perfectly acceptable to have the NICs unteamed, one attached to a vSwitch and the other used for host networking traffic. When using this configuration, one NIC will use VMQ and the other will use RSS.
VMQ leverages the same hardware queues on the NIC as RSS, the ones that interrupt different cores. However with VMQ the filters and logic associated with queue distribution of the packets is different. Rather than one physical device utilizing all the queues for its networking traffic, these queues are balanced among the host and VMs on the system.
Inhaltsverzeichnis
Important / to consider
- Do not use Core 0! (Because of the Parent Partition (Management OS),...)
- Do not use Hyperthreaded Cores!
- NUMA-Distance should be 0!
- To fully utilize the CPUs, the number of RSS Receive Queues should be equal to or greater than Max Processors!
RSS
Receive Side Scaling (RSS) is a very important aspect in networking on Windows. RSS makes sure that incoming network traffic is spread among the available processors in the server for processing. If we do not use RSS, network processing is bound to one processor which will limit approximately at 4GBps. Nowadays every NIC has RSS enabled by default, but the configuration is not optimized. Every NIC is configured with “Base Processor” 0, meaning it will start processing on processor 0 together with the others NICs but more importantly it’s the default processor for Windows processes as well.
To optimally configure RSS we want to start at processor 1 so we don’t interfere with processes landing default on processor 0. On the other hand, we also want to keep the NUMA node assignment that RSS traffic is not using a processor in NUMA node 0 while the NIC is bound to NUMA node 1.
RSS Profiles
Closest: Logical processor numbers near the network adapter’s base RSS processor are preferred. Windows may rebalance logical processors dynamically based on load.
ClosestStatic: No dynamic load balancing, such as distributing but not load balancing at runtime.
NUMA: Assigns RSS processors in a round robin basis across every NUMA node to enable applications that are running on NUMA servers to scale well.
NUMAStatic: Default behavior. RSS processor selection is the same as for NUMA scalability without dynamic load balancing.
Conservative: RSS uses as few processors as possible to sustain the load. This option helps reduce the number of interrupts.
VMQ
With the entrance of virtualization on servers and RSS we hit some limitations because RSS wasn’t built for this “new world”. Long story short, when having a virtual switch on the system where physical adapters are bound to, RSS is disabled. This caused scalability issues that needed to be solved… hello VMQ.
The real benefit of VMQ is realized when it comes time for the vSwitch to do the routing of the packets. When incoming packets are indicated from a queue on the NIC that has a VMQ associated with it, the vSwitch is able the direct hardware link to forward the packet to the right VM very very quickly – by passing the switches routing code. This reduces the CPU cost of the routing functionality and causes a measurable decrease in latency.
Unlike RSS where incoming traffic is spread among processors, VMQ will create a queue for each mac-address on the system and link that queue to a processor core. This means that a mac-address, which translates to a NIC of a VM or a host vNIC does not have more process power than 1 processor core. This limits the network traffic speed in the same way it does when we do not enable RSS to about 4GBps.
VMQ-Settings setzen
# VMQ aktivieren Set-NetAdapterVMQ -Name <NIC-Name> -Enabled # RSS aktivieren Set-NetAdapterRss -Name <NIC-Name> -Enabled # VMQ-Settings setzen Set-NetAdapterVmq -Name <NIC-Name> -BaseProcessorNumber <erster Core, nicht Core 0> -MaxProcessors <anzahl Cores> -MaxProcessorNumber <letzer Core> -NumaNode <NUMA-Node>
Die Anzahl Queues können ebenfalls mit dem Parameter -NumberOfReceiveQueues gesetzt werden. Hier wird empfohlen die Einstellungen auf Automatisch zu belassen.
Wenn genug Cores pro CPU vorhanden sind wird für eine 1Gig NIC -> 8 Queues empfohlen und für eine 10Gig NIC und höher -> 16 Queues.
RSS-Settings Beispiel (optimal, bei insgesamt 32 logischen Cores)
Wir haben einen Server mit folgender Hardware:
- 2 CPUs (sprich 2 Sockets) mit je 8 Cores (+ Hyperthreading aktiviert = 16 logische Cores pro NUMA-Node)
- 2 NICs (in einem Team/Bond)
- Number of Receive Queues = 127
Die Optimale Verteilung der Queues wäre folgende:
Blau markiert die physischen Cores.
Es dürfen keine Hyperthreadete Cores verwendet werden!
Die erste NIC verteilen wir auf den ersten NUMA-Node (NUMA-Node 0), aber wir lassen Core 0 aus, sprich der erste Core im NUMA-Node 0)
Die zweite NIC verteilen wir auf den zweiten NUMA-Node (NUMA-Node 1), aber wir lassen Core 0 aus, sprich der erste Core im NUMA-Node 1)
RSS-Settings setzen
Set-NetAdapterRss -Name NIC01 -Enabled Set-NetAdapterRss -Name NIC02 -Enabled Set-NetAdapterRss -Name NIC01 -BaseProcessorNumber 2 -MaxProcessors 7 -MaxProcessorNumber 14 -NumaNode 0 Set-NetAdapterRss -Name NIC02 -BaseProcessorNumber 18 -MaxProcessors 7 -MaxProcessorNumber 30 -NumaNode 1
RSS-Settings Beispiel (optimal, bei insgesamt 56 logischen Cores)
Wir haben einen Server mit folgender Hardware:
- 2 CPUs (sprich 2 Sockets) mit je 14 Cores (+ Hyperthreading aktiviert = 28 logische Cores pro NUMA-Node)
- 2 NICs (in einem Team/Bond)
- Number of Receive Queues = 127
Die Optimale Verteilung der Queues wäre folgende:
Blau markiert die physischen Cores.
Es dürfen keine Hyperthreadete Cores verwendet werden!
Die erste NIC verteilen wir auf den ersten NUMA-Node (NUMA-Node 0), aber wir lassen Core 0 aus, sprich der erste Core im NUMA-Node 0)
Die zweite NIC verteilen wir auf den zweiten NUMA-Node (NUMA-Node 1), aber wir lassen Core 0 aus, sprich der erste Core im NUMA-Node 1)
RSS-Settings setzen
Set-NetAdapterRss -Name NIC01 -Enabled Set-NetAdapterRss -Name NIC02 -Enabled Set-NetAdapterRss -Name NIC01 -BaseProcessorNumber 2 -MaxProcessors 13 -MaxProcessorNumber 26 -NumaNode 0 Set-NetAdapterRss -Name NIC02 -BaseProcessorNumber 30 -MaxProcessors 13 -MaxProcessorNumber 54 -NumaNode 1
More Informations:
RSS: https://docs.microsoft.com/en-us/windows-hardware/drivers/network/introduction-to-receive-side-scaling
VMQ Best Practise: https://blogs.technet.microsoft.com/networking/2016/01/04/virtual-machine-queue-vmq-cpu-assignment-tips-and-tricks/
Optimizing Network: http://www.darrylvanderpeijl.com/windows-server-2016-networking-optimizing-network-settings/
Other Offloading Features: https://msdn.microsoft.com/en-us/library/windows/hardware/dn567650(v=vs.85).aspx
SAP: https://blogs.msdn.microsoft.com/saponsqlserver/2012/01/11/network-settings-network-teaming-receive-side-scaling-rss-unbalanced-cpu-load/
