Monitoring Io Usage: Linux Tips And Tricks

Monitoring input/output (I/O) usage is an important aspect of maintaining optimal performance in Linux systems. High disk I/O operations can lead to server performance degradation, causing applications to slow down.

There are several command-line tools available in Linux distros that can help monitor and troubleshoot disk I/O activity:

- iostat: Reports disk read/write rates and counts at continuous intervals. It can also be used to monitor CPU statistics and input/output statistics for devices and partitions.

- iotop: Similar to the 'top' command, iotop displays real-time disk activity and the processes that are performing I/O, along with the disk bandwidth they are using.

- dstat: A user-friendly version of iostat that can show CPU and disk stats, disk utilisation, and transactions (I/O requests) per second.

- atop: Provides an overview of system performance, including CPU usage, memory, swap file usage, and disk and network information. It can also list processes that caused system-level changes during a specific interval.

- ioping: A quick tool to check storage volume latency, useful for diagnosing disk problems.

Using the iostat command

The iostat command is a valuable Linux command-line utility that provides detailed information about your system's CPU and disk I/O performance. It is part of the sysstat package and offers real-time insights into the utilisation of your system's resources, allowing you to identify bottlenecks and optimise performance.

The basic syntax of the iostat command is as follows:

Iostat [options] [devices] [interval] [count]

• Options: Flags that modify the command's behaviour or specify the type of information to display.
• Devices: The devices for which you want to display I/O statistics. By default, iostat shows statistics for all devices.
• Interval: The time interval (in seconds) between updates. By default, iostat displays a single report.
• Count: The number of updates to display before exiting. By default, iostat will continue running indefinitely if an interval is specified.

To display basic CPU and disk I/O statistics, simply run the command without any options. The output of the iostat command is divided into two sections: CPU statistics and device statistics.

• %user: The percentage of time the CPU spent executing user-level processes.
• %nice: The percentage of time the CPU spent executing user-level processes with a positive nice value.
• %system: The percentage of time the CPU spent executing system-level processes.
• %iowait: The percentage of time the CPU spent waiting for I/O operations to complete.
• %steal: The percentage of time the CPU spent in involuntary wait while the hypervisor serviced another virtual processor.
• %idle: The percentage of time the CPU was idle.
• Device: The name of the device (e.g. sda, sdb).
• Tps: The number of transfers per second (I/O requests) sent to the device.
• KB_read/s: The number of kilobytes read from the device per second.
• KB_wrtn/s: The number of kilobytes written to the device per second.
• KB_read: The total number of kilobytes read from the device.
• KB_wrtn: The total number of kilobytes written to the device.
• Monitor I/O statistics for specific devices (e.g. sda and sdb) every 2 seconds: iostat -d 2 sda sdb
• Display extended disk I/O statistics: iostat -x
• Display only CPU-related statistics: iostat -c
• Display I/O statistics for all devices and include a timestamp for each report: iostat -t
• Display the average statistics since the system was last booted: iostat -av
• Monitor I/O statistics for specific devices every 5 seconds for a total of 10 updates: iostat -d 5 10 sda sdb

High values in certain metrics can indicate performance issues or bottlenecks. For example:

• High %iowait: Indicates that the CPU is waiting a significant amount of time for I/O operations to complete, possibly due to an I/O bottleneck.
• High tps: Suggests heavy I/O activity on the disk.
• High kB_read/s and kB_wrtn/s: Indicates that the disk is reading and writing a large amount of data, potentially causing an I/O bottleneck.
• High %user and %system: Suggests that the CPU is spending a lot of time executing user and system processes, which may require optimisation or CPU upgrade.

Using the iotop command

The iotop command is a top-like utility for monitoring disk input and output (I/O). It is used to see I/O usage information output by the Linux kernel. The iotop command displays a table of current I/O usage by Linux processes or threads on the Linux system.

To use the iotop command, you must first install the iotop utility. Here are the commands for common Linux distros:

• Installing the iotop on a CentOS/RHEL/Fedora Linux: Use the yum command or dnf command as follows: # yum install iotop
• Debian/Ubuntu Linux install iotop: Use the apt command or apt-get command as follows: $ sudo apt-get install iotop
• OpenSUSE/SUSE Linux install iotop: Run the zypper command: $ sudo zypper in iotop

To run iotop, use the following command (must be run as root): $ sudo iotop OR # iotop

The iotop command displays columns for the I/O bandwidth read and written by each process/thread during the sampling period. It also displays the percentage of time the thread/process spent while swapping in and while waiting on I/O. For each process, its I/O priority (class/level) is shown.

In addition, the total I/O bandwidth read and written during the sampling period is displayed at the top of the interface.

• To start the iotop I/O monitor, use: $ iotop
• To show only processes or threads actually doing I/O, use: $ iotop -o
• To show I/O usage in non-interactive mode, use: $ iotop -b
• To show only I/O usage of processes (default is to show all threads), use: $ iotop -P
• To show I/O usage of given PID(s), use: $ iotop -p PID
• To show I/O usage of a given user, use: $ iotop -u USER
• To show accumulated I/O instead of bandwidth, use: $ iotop -a
• To analyze the performance of a single program (PID) and log the results to a file, use: $ iotop -bp PID > iostats.log

These are some basic examples of using the iotop command. For more advanced usage, you can refer to the iotop manual page or other online resources.

Using the atop command

The `atop command is a useful tool for monitoring disk I/O usage in Linux systems. It provides detailed information about system performance, including disk activity stats.

To use `atop`, simply enter the following command:

Bash

Atop

This will display a comprehensive overview of system performance, including process information, CPU usage, load averages, memory and swap usage, and disk and network information.

To focus specifically on disk stats, you can use the following command:

Bash

Atop | grep DSK

This will filter the `atop` output to only show lines containing "DSK", which pertain to disk activity.

Additionally, `atop` can be used to monitor system changes over time intervals. For example, to run `atop` with a one-second interval, use:

Bash

Atop -i 1

This will update the `atop` output every second, allowing you to see changes in real time.

`atop` is particularly useful for identifying processes that are causing high disk I/O activity. By default, it displays a view of disk I/O by each process or thread.

To only show processes or threads that are actually performing I/O activity, use the `-o` option:

Bash

Atop -o

This will help you pinpoint specific processes that may be causing performance issues related to disk I/O usage.

Furthermore, `atop` allows you to save the output to a file for later analysis. To do this, use the `-w` option to specify the output file:

Bash

Atop -w output_file.atop

This will write the `atop` output to the specified file, which can then be reviewed at a later time using the `-r` option:

Bash

Atop -r output_file.atop

By utilizing the `atop` command and its various options, you can effectively monitor and analyze disk I/O usage in your Linux system, helping you identify and address any performance issues related to disk activity.

Using the dstat command

The dstat command is a useful tool for monitoring disk I/O in Linux. It provides insights into overall system performance, including disk activity, CPU usage, network and paging activity, interrupts, and context switches.

To focus solely on disk activity, the -d option can be used. This will display continuous measurements of disk read and write operations until the display is stopped. The default interval for each subsequent row in the display is one second, but this can be adjusted by including a number after the -d option, specifying the interval in seconds. For example:

$ dstat -d 10 -dsk/total

Read writ

949B 73k

65k 81M

The dstat command also allows for the specification of exact devices to report on, using the -D option. This can be useful for monitoring individual disk I/O:

$ dstat -tdD total,sda,sdb,sdc,md1 60

System---- -dsk/total----dsk/sda-----dsk/sdb-----dsk/sdc-----dsk/md1--

Time | read writ: read writ: read writ: read writ

08-11 22:08:17|3549k 277k: 144k 28k: 851k 62k: 852k 60k: 25k 82k

Additionally, the --disk-util option can be used to display disk utilisation, and --disk-tps for disk transactions (I/O requests) per second.

Using the ioping command

The ioping command is a simple disk I/O latency monitoring tool. It can be used to measure the time it takes for a disk to respond to requests, which can be helpful in diagnosing disk problems.

To use ioping, you can specify a directory, file, or device as an argument. For example, to monitor the I/O latency of your current directory, you can simply run the command `ioping .` You can also specify the number of requests to make using the `-c` option, such as `ioping -c 10 .` to make 10 requests.

The ioping command also supports various options to customize its behaviour:

• The -`i` option allows you to set the interval between requests in seconds.
• The -`s` option lets you specify the request size in bytes.
• The -`l` option sets a speed limit in bytes per second.
• The -`r` option sets a rate limit in requests per second.
• The -`t` and -`T` options specify the minimum and maximum valid request times, respectively.
• The -`o` option sets the starting offset in the file or device.
• The -`w` option stops the command after a certain amount of time has passed.
• The -`p` and -`P` options print raw statistics at specific intervals or after a certain number of requests.
• The -`A` option enables asynchronous I/O.
• The -`C` option enables cached I/O without cache flush or drop.
• The -`D` option enables direct I/O.
• The -`R` option performs a disk seek rate test.
• The -`L` option uses sequential operations instead of random ones.
• The -`W` option uses write I/O, which can be more reliable for systems that don't support non-cached reads.
• The -`G` option enables read-write ping-pong mode, alternating between read and write requests.
• The -`Y` option enables sync I/O.
• The -`y` option enables data sync I/O.
• The -`k` option keeps and reuses a temporary working file.
• The -`q` option suppresses human-readable output.
• The -`h` option displays a help message.
• The -`v` option displays the version of ioping.
• `sudo ioping /dev/sda1 4 KiB` — Monitor the I/O latency of the `/dev/sda1` device with a request size of 4 KiB.
• `ioping -c 10 -s 1M /tmp` — Make 10 requests of 1 megabyte each to measure the latency on the `/tmp` directory.
• `ioping -R /dev/sdX` — Measure the disk seek rate on the `/dev/sdX` device.
• `ioping -RL /dev/sdX` — Measure the disk sequential speed on the `/dev/sdX` device.

Frequently asked questions