Monitoring Linux Server Performance: Key Metrics And Methods

Linux is one of the most well-known and widely used open-source operating systems. It is stable, secure, compatible, and customisable.

Linux servers form the backbone of any IT infrastructure, so it is crucial to monitor and maintain them to ensure optimal performance and seamless operations of the applications that rely on them.

- CPU Usage: Regularly monitoring CPU usage will help analyse the CPU load over a given period and overcome performance regressions.

- System Load: System load is a measurement of the computational work the system is performing. It represents the average number of instructions waiting for CPU time and the overall demand for CPU.

- Memory Utilisation: Monitoring memory helps identify over- and underused servers and redistribute loads effectively.

- Disk Utilisation: Tracking disk utilisation helps maintain the efficiency of a Linux system and minimise the risk of hardware failure.

CPU Utilization

• Top Command: The top command displays real-time information about CPU usage, memory, and running processes. It provides an overview of the system's performance, including the percentage of CPU time used by each process.
• Mpstat Command: Mpstat is part of the sysstat package and is used to display CPU activity on a per-processor basis. It provides detailed statistics such as user, system, and idle time for each processor or processor core.
• Sar Command: The sar command is a versatile tool for monitoring system resources. By using the -u option, you can track CPU performance and view CPU usage over time.
• Iostat Command: The iostat command displays information about CPU utilization, device utilization, and network file system utilization. It provides insights into average CPU usage since the last system reboot.
• Vmstat Command: The vmstat command shows system processes, memory, swap, I/O, and CPU performance. It provides average statistics since the last reboot and can be used to monitor CPU usage over time.
• Nmon: Nmon is a monitoring tool developed by IBM that provides insights into CPU usage, memory, disk usage, and more. It offers both online and capture modes for real-time monitoring and data collection, respectively.

Load Average

The load average is calculated by averaging the number of active and waiting tasks over the past 1, 5, or 15 minutes. The metric is expressed as the average number of processes in a runnable state over these time periods, with a higher load average indicating higher resource usage.

A normal load average depends on the number of CPUs and cores in the system. Dual-CPU systems can handle more concurrent processes, resulting in a lower load average. In contrast, a high load average with a low number of cores indicates an overwhelmed system. Generally, a load average of less than the number of CPU cores is normal, as it means there are enough resources for all processes to run smoothly. For example, in a quad-core CPU system, a load average of less than 4 is considered normal.

To check the load average in Linux, you can use commands such as `uptime`, `top`, `htop`, `cat /proc/loadavg`, `w`, and `glances`. These commands provide insights into the load average and other system performance metrics.

A high load average indicates that the system has a higher number of tasks being processed or waiting in the queue, signalling higher CPU resource usage. This can be addressed by optimising code, running processes in parallel, increasing the number of CPU cores, or scheduling tasks to run at different times.

A low load average, on the other hand, suggests that the system has spare capacity and can handle more processes. However, a load average of 0 does not necessarily mean the system is idle, as it may still be running background processes or small jobs.

Memory Utilization

Memory utilisation is a key metric to monitor when it comes to Linux server performance. Memory usage can be monitored using a variety of tools and commands, which provide valuable insights into the memory consumption of the system.

Free Command

The `free` command is a simple and widely used method to check memory usage on Linux. It displays the total amount of RAM installed on the system, along with the used and free memory. The -m option presents the output in megabytes, making it easier to interpret. The `free` command also provides information about swap memory, which is used when physical memory is full.

/proc/meminfo File

Another way to check memory usage is by reading the /proc/meminfo file. This virtual file contains dynamic information about the system's memory usage, including total memory, free memory, buffers, and cached memory. It offers a detailed breakdown of memory-related statistics.

Vmstat Command

The `vmstat` command provides comprehensive insights into virtual memory statistics. With the -s option, `vmstat` displays memory usage details such as total memory, used memory, active memory, inactive memory, free memory, buffer memory, and swap cache.

Top and htop Commands

The `top` command is a versatile tool that displays CPU and memory usage, as well as providing a list of running processes. It shows the total, used, and free memory, along with buffer and cache information. Additionally, the `htop` command offers a similar functionality with an enhanced user interface and additional features for process management.

Dmidecode Command

For hardware-related information about the installed RAM, the `dmidecode` command can be used. It provides details such as the size, type, and speed of the RAM modules. This command requires root privileges to execute.

By utilising these commands and tools, system administrators can effectively monitor memory utilisation on Linux servers, ensuring optimal performance and resource allocation.

Disk Utilization

To check disk utilization, you can use the following commands:

• Df (disk free): This command displays the amount of available disk space on file systems. You can use the -h option to display sizes in a human-readable format, such as KB, MB, and GB. You can also specify a particular filesystem to get information about, such as /dev/sda1.
• Du (disk usage): This command estimates file space usage and provides detailed information about the disk space used by files and directories. You can use the -h option to display sizes in a human-readable format and -s to summarize the total for each argument.
• Pydf: This is a Python-based command that provides a colorful and user-friendly output of disk space usage. It is designed as an alternative to the df command.
• Fdisk -l: While the fdisk command primarily displays disk partition details, it also includes information about disk sizes and provides general information about disk capacity.
• Lsblk: This command lists information about block devices, including their names, sizes, types, and mount points. It provides a hierarchical view of block devices but does not give detailed disk usage information.

By regularly monitoring disk utilization, you can ensure that your Linux system has sufficient free space to operate efficiently and prevent potential issues caused by low disk space.

I/O Wait Time

The I/O wait time is the duration between your input and the output provided by the system. While this may seem harmless, over time, it can drain the kernel and slow down the CPU, resulting in poor performance. High levels of I/O wait can indicate that the computer's I/O subsystem is a bottleneck, and the CPU is not being utilised efficiently.

There are several reasons why a CPU may experience high I/O wait times:

• Disk I/O: The computer is reading from or writing to a storage device, and the CPU waits for the data transfer to be completed.
• Network I/O: The computer is waiting for data transfer over a network connection.
• Other I/O operations: The computer is waiting for other I/O operations, such as reading from or writing to a USB device or printer.
• Insufficient CPU resources: The CPU is unable to keep up with the demand for computational tasks, resulting in significant I/O wait time.
• Slow I/O subsystem: A slow or overloaded I/O subsystem prolongs the completion of I/O operations, leading to higher I/O wait times.

To assess high I/O wait times in Linux, you can use the following commands:

• `top`: This command displays a real-time view of system performance, including CPU-level statistics and a list of running processes. The I/O wait time is denoted by `wa` in the `%Cpu(s)` line.
• `vmstat`: This command provides information about memory, processes, CPU, and more. It displays the I/O wait time and additional data such as total free memory.
• `iotop`: This command is similar to `top` but focuses on monitoring disk I/O usage and the associated running processes or threads.
• `iostat`: This command displays system input/output statistics, device-level reports, and CPU statistics. It helps identify the disks involved in I/O operations.

To fix high I/O wait times, you can try the following strategies:

• Identify the root cause of the issue using system monitoring tools like `top` and `iotop`.
• Check for disk errors and fix them using the `fsck` command.
• Tune your I/O scheduler by changing the `elevator` kernel parameter.
• Increase the number of simultaneous I/O requests by adjusting the `nr_requests` kernel parameter.
• Optimise your disk layout by placing frequently accessed files on faster disks or near the beginning of the disk.

Frequently asked questions