Monitoring Hyper-V Performance: A Comprehensive Guide

Hyper-V performance monitoring is essential to ensure optimal hardware resource usage and detect bottlenecks. By monitoring Hyper-V performance, administrators can make informed decisions on resource allocation and configuration adjustments. This includes understanding the requirements of the workloads, identifying software dependencies, and categorising resource utilisation parameters such as memory, CPU, storage options, and network access. While tools like Task Manager and Performance Monitor are useful for basic monitoring, they may not provide a comprehensive view of performance in a virtualised environment. Hyper-V-specific tools and counters are available to monitor CPU usage, memory availability, disk latency, and network performance, helping to identify and address potential bottlenecks and performance issues.

Monitor CPU, memory, disk read/write and network utilisation

Monitoring the CPU, memory, disk read/write and network utilisation of Hyper-V is crucial for optimising performance and resource allocation. Here are some detailed instructions and tips for achieving this:

CPU Utilisation

Use the Hyper-V Hypervisor Logical Processor% Total Run Time counter to monitor physical processor performance on a Hyper-V host. This counter provides information on the percentage of time the processor is active. If this value exceeds 90%, the host is overloaded, and you should consider adding more processing power or redistributing virtual machines.

To monitor the performance of virtual processors within a VM, use the Hyper-V Hypervisor Virtual Processor% Guest Run Time counter. This counter displays the percentage of time the virtual processor is active. Optimal performance is indicated by a value of 75% or less. A range of 75-85% is a warning, and anything above 85% warrants investigation.

Memory

The MemoryAvailable Mbytes counter on a Hyper-V host is used to monitor memory usage. A healthy system will have over 10% of RAM available, while values below 10% are a warning, and anything under 100 MB is critical.

Additionally, use the MemoryCommitted Bytes counter to determine how much memory each VM is utilising to make informed decisions about memory allocation for each VM.

Disk Read/Write

The \LogicalDisk(*)\Average Disk Sec\Read or Write counters are essential for monitoring disk latency, which significantly impacts the overall performance of VMs.

Healthy disk latency is indicated by values less than 10 ms (0.010) s. A value above 15 ms (0.015) s warrants attention, and anything above 25 ms (0.025) s is critical and requires immediate action.

Network Utilisation

Monitor network usage with the \Network Interface (*)\OutputQueue Length counter, which measures the number of network packets in the output packet queue.

A healthy network has an average value of less than 1. A value above 1 on average is a warning, and if the average exceeds 2, it is considered critical.

Additionally, you can use the following counters for further insights:

• Network Interface (network adapter name)\Bytes/sec
• Hyper-V Virtual Network Adapter (virtual machine name)\Bytes/sec

Monitor processor performance

Monitoring processor performance is crucial to ensure optimal resource allocation and prevent performance degradation in Hyper-V environments. Here are some detailed instructions and best practices for monitoring processor performance in Hyper-V:

Hyper-V Logical Processor Performance Monitoring:

The Hyper-V Logical Processor counter measures the actual CPU usage on the server. To monitor this:

• Go to the Start Button, then Administrative Tools, and select Performance Monitor.
• Expand the Monitoring Tools option and select Performance Monitor again.
• To view the Total Run Time of the Logical Processor, click the Add (+) icon.
• In the new window, select the HyperV server from the dropdown list or browse for it if not found.
• Choose the Hyper-V Hypervisor Logical Processor option and click the + icon to view sub-options.
• Select % Total Run Time and click Add>>.
• Now you can monitor and observe the graph for this counter.

Interpret the results as follows:

• Less than 60% CPU usage – Healthy state.
• Between 60% and 89% CPU usage – Monitor or Caution state.
• Above 90% CPU usage – Critical state, indicating potential performance issues.

Hyper-V Virtual Processor Performance Monitoring:

The Hyper-V Virtual Processor counter helps identify Virtual Machines (VMs) consuming the most CPU resources. To monitor this:

• Follow steps 1-5 from above, then select Hyper-V Hypervisor Virtual Processor and click the + icon.
• Choose % Guest Run Time and select the option to view all VMs.
• Click Add>> and then OK to start monitoring.
• Identify the VMs with high CPU consumption and investigate further.

Performance Counter Interpretation:

When interpreting the performance counters, it is important to understand the thresholds and take appropriate actions:

• Optimal performance is indicated by a counter value of 75% or less for a running VM.
• A value between 75% and 85% is a warning, indicating potential CPU bottlenecks.
• A value higher than 85% is critical, and you should investigate the reasons for high CPU usage.

Additionally, if the Hyper-V Hypervisor Logical Processor(_Total)\% Total Runtime counter exceeds 90%, it suggests that the host is overloaded, and you should consider adding more processing power or redistributing VMs across different hosts.

Other Recommendations:

• Avoid using Windows Task Manager to monitor CPU usage inside VMs, as it does not accurately reflect the actual CPU usage due to its inability to identify virtual machines.
• Utilize Hyper-V-specific counters in the Performance Monitor to get the most accurate information for Hyper-V processor performance monitoring.
• Regularly monitor and optimize processor performance to ensure efficient resource allocation and maintain the overall health of your Hyper-V environment.

Monitor memory

Monitoring memory usage is crucial to ensure optimal performance of Hyper-V virtual machines (VMs). Insufficient memory allocation can lead to performance degradation, while excess allocation can impact the performance of other VMs and the entire system. Here are some detailed instructions and tips for monitoring memory usage in Hyper-V:

Tools for Monitoring Memory

• Performance Monitor (perfmon): This is a widely used utility for troubleshooting performance on Windows systems. To monitor memory usage on a Hyper-V host, run perfmon and add the necessary counters, such as MemoryAvailable Mbytes and MemoryCommitted Bytes. These counters will help you understand the memory usage of the Hyper-V host and individual VMs.
• Resource Monitor (resmon.exe): Opening Resource Monitor on a Hyper-V host can provide additional insights into memory utilisation.
• Event Viewer: Event Viewer allows you to monitor logs on Hyper-V hosts and guest VMs, which can be useful for troubleshooting. To access Hyper-V-specific logs, go to Event viewer > Applications and services logs > Microsoft > Windows.
• PowerShell: For non-GUI Windows OS, PowerShell can be used to monitor resources of Hyper-V hosts and VMs. You can use special cmdlets and commands to interact with performance counter objects and retrieve data directly from local or remote machines.
• Third-Party Tools: If the native tools do not meet your requirements, consider using third-party software for Hyper-V monitoring, such as SCVMM (System Center Virtual Machine Manager) or Hyper9.

Memory Bottlenecks and Optimisation

To optimise memory usage and avoid bottlenecks, consider the following:

• Provide Enough Memory for VMs: Ensure that each VM has sufficient memory allocated for proper functioning. For example, if an application running on a VM consumes 6 GB of RAM, it is recommended to provide 8 GB of RAM for that VM, with at least 20% additional memory as a buffer.
• Monitor Memory Counters: Keep an eye on memory-related performance counters, such as Memory\Available Mbytes and Hyper-V Dynamic Memory Balancer ()\Available Memory. If these counters are low on the host, consider stopping non-essential services and migrating VMs to another host. If they are low on a VM, assign more memory to that VM.
• Dynamic Memory: Utilise the dynamic memory feature in Hyper-V to automatically adjust memory allocation based on demand. Monitor the \Hyper-V Dynamic Memory Balancer\Average Pressure counter to keep track of dynamic memory usage.
• Identify Memory-Intensive VMs: Identify VMs that are consuming a significant amount of memory and consider optimising their memory usage or migrating them to another host to balance the load.
• Monitor Disk Latency: High disk latency can impact overall VM performance. Monitor the \LogicalDisk()\Average Disk Sec\Read and Write counters to keep track of disk latency. If the latency is consistently above 25 ms, consider optimising storage configurations.

Monitor network performance

Monitoring network performance is crucial to ensure optimal performance of Hyper-V virtual machines. Here are some methods and best practices to monitor network performance:

• Test network latency: Ping each virtual machine to ensure adequate network latency. On local area networks, expect response times of less than 1ms.
• Test for packet loss: Use tools such as pathping.exe to test packet loss between virtual machines. This utility measures packet loss and can help identify network issues.
• Test network file transfers: Copy a file of a certain size (e.g., 100MB) between virtual machines and measure the time taken. On a healthy network, this transfer should take 10-20 seconds on a 100Mbit network and about 3-5 seconds on a 1Gbit network.
• Monitor network utilisation: Use performance monitor counters such as "\Network Interface()\Bytes Total/sec" to measure network utilisation on the Hyper-V host operating system. Evaluate network bandwidth utilisation and take appropriate action if it exceeds healthy thresholds.
• Monitor output queue length: Measure the number of threads waiting on the network adapter by using the "\Network Interface()\Output Queue Length" counter. If there are more than two threads waiting, the network may be a bottleneck, and you should consider optimisations or adding more physical network adapters.
• Use Hyper-V Virtual Network Adapter: If the network adapter on the Hyper-V root partition is busy, use the "\Hyper-V Virtual Network Adapter()\Bytes/sec" performance monitor counter to identify which virtual network adapters are consuming the most network resources.
• Optimise processor performance: Balance the values reported by the host operating system for "Hyper-V Hypervisor Logical Processor(_Total)\% Total Run Time" (LPTR) and "Hyper-V Hypervisor Virtual Processor(_Total)\% Total Run Time" (VPTR). Ensure that there are not more processors allocated to virtual machines than physically available.
• Use alerts and dashboards: Utilise tools that offer customisable dashboards and alerts to monitor network performance. For example, SolarWinds Virtualization Manager provides alerts for critical issues like high CPU utilisation and network latency.

Monitor storage

Storage bottlenecks can cause host and virtual machine operations to slow down or time out, or for the virtual machine to become sluggish. To monitor storage, you can use the following performance counters from the host:

• Physical Disk(disk letter)\Avg. disk sec/Read
• Physical Disk(disk letter)\Avg. disk sec/Write
• Physical Disk(disk letter)\Avg. disk read queue length
• Physical Disk(disk letter)\Avg. disk write queue length

If latencies are consistently greater than 50ms, you should spread virtual machines across additional storage, consider purchasing faster storage, or consider Tiered Storage Spaces or Storage QoS.

Virtual Fibre Channel HBAs

Virtual Fibre Channel HBAs allow VMs to directly access Fibre Channel and Fibre Channel over Ethernet (FCoE) logical unit numbers (LUNs). Virtual Fibre Channel disks bypass the NTFS file system in the root partition, which reduces storage I/O central processing unit (CPU) usage. They are great for large data drives and drives shared between multiple VMs in guest clustering scenarios.

To use Virtual Fibre Channel disks, you must install one or more Fibre Channel HBAs on the host machine. Each host HBA must use HBA drivers that support Windows Server 2016 Virtual Fibre Channel or N_Port ID Virtualization (NPIV) capabilities. The Storage Area Network (SAN) fabric should also support NPIV, and you should configure the HBA ports for the Fibre Channel in a Fibre Channel topology that supports NPIV.

To maximize throughput on hosts with more than one HBA, configure multiple virtual HBAs inside the Hyper-V VM. You can configure up to four HBAs for each VM. Hyper-V automatically balances virtual HBAs to host HBAs accessing the same virtual SAN.

Virtual Disks

Virtual disks are exposed to the VMs by virtual controllers and can be virtual hard disks or pass-through disks on the host. They come in VHD or VHDX formats, with each format supporting three types of virtual hard disk files.

• VHD – If you upgrade your deployment to Windows Server 2016 or later, it is recommended that you convert all VHD files to VHDX format. Later versions of Hyper-V include improvements to their VHD format that allow for better alignment.
• VHDX – An updated hard disk format introduced in Windows Server 2012 that can create resilient, high-performance virtual disks with up to 64 terabytes of capacity. It also provides several performance features, such as improved alignment of the virtual hard disk format, larger block sizes for dynamic and differential disks, a 4-KB logical sector virtual disk, efficiency in representing data, and trimming capabilities.

There are three types of VHD files:

• Fixed files – Improve resiliency and performance and should be used when storage on the hosting value isn't actively monitored. They are less likely to fragment, reducing the I/O throughput, and have the lowest CPU overhead.
• Dynamic files – Provide resiliency guarantees and allocate disk space as the deployment needs it. They are only available on VHDX.
• Differencing files – Snapshots of a VM that store writes to the disks. They keep VM snapshot chains short to maintain good disk I/O performance. They can be used on any disk format.

Storage QoS

Storage QoS (quality of service) is available in Windows Server 2012 R2 and later, allowing you to set certain QoS parameters for storage on VMs. You can configure storage performance isolation in a multitenant environment, specify the maximum and minimum input/output operations per second (IOPS) for virtual hard disks, and monitor disk performance.

NUMA I/O Registry Settings for Large VMs

Windows Server 2012 and later support projecting a virtual, nonuniform memory access (NUMA) topology into Hyper-V VMs, improving performance for workloads running on large VMs. To enable this support, large VM configurations require scalability in terms of I/O throughput.

To improve the performance of workloads running on large VMs, use the Windows Server NUMA registry key settings. You can adjust the number of channels using the following registry entries:

• ChannelCount (DWORD) – The total number of communication channels that can be used. The maximum value is 16.
• ChannelMask (QWORD) – The processor affinity for the channels. If not specified, the channel mask defaults to the existing channel distribution algorithm for normal storage or networking channels.

Offloaded Data Transfer Integration

Offloaded Data Transfer (ODX) operations allow the VM workload to use ODX-enabled storage as it would in a physical environment. This feature provides near-instantaneous copy operations for large amounts of data and allows the system to specify how to move specific data sets from one location to another.

Unmap Notification Integration

Unmap notifications make VHDX files more efficient and allow the underlying physical storage device to reclaim unused space. This feature lets VHDX files report deleted data to the storage stack, keeping file sizes trim and allowing the stack to reclaim unused storage space for other uses.

Frequently asked questions