Monitoring Gpu Usage: Linux Command-Line Essentials

Monitoring GPU usage on Linux is an important task for those who want to keep track of their system's performance and ensure that their GPU is being utilized efficiently. Here are some methods to monitor GPU usage in Linux:

- nvidia-smi: This is a command-line utility provided by NVIDIA that allows users to monitor the performance of their NVIDIA GPU devices. It provides information such as memory usage, GPU utilization, and temperature.

- nvtop: A ncurses-based GPU status viewer for NVIDIA GPUs, similar to the htop or top commands. It displays real-time GPU and GPU memory usage, temperature, power usage, and graph information.

- intel-gpu-tools: This package includes the intel_gpu_top command, which provides a top-like summary of Intel GPU usage, including real-time IMC read and write speed, power usage, and percentage usage of 3D/Render, Blitter, Video, and VideoEnhance.

- radeontop: A tool to show AMD GPU utilization, including total activity percentage and individual block usage. It works with R600 and newer GPUs and is compatible with both open-source AMD drivers and AMD Catalyst closed-source drivers.

- atop: A UNIX command-line utility that monitors system resources and performance, including CPU, GPU, memory, disk, and network usage. It logs system activity in 10-minute intervals, allowing users to review GPU usage over time.

- glances: A monitoring program with a GPU monitoring plugin that displays GPU utilization and memory usage, along with other system information such as CPU, disk IO, disk space, and network usage.

- Mission Control: A graphical app that provides an interface similar to Windows Task Manager for monitoring system resource usage. It supports overall GPU usage monitoring, video encoder and decoder usage, memory usage, and power consumption for AMD, Apple, Huawei, Intel, NVIDIA, and Qualcomm GPUs.

Using the nvidia-smi command

The NVIDIA System Management Interface (nvidia-smi) is a command-line utility based on the NVIDIA Management Library (NVML) designed to help manage and monitor NVIDIA GPU devices. It is installed with the CUDA toolkit and provides meaningful insights into your GPU's performance.

The nvidia-smi command can be used to monitor GPU performance, including tracking GPU and memory utilisation, and temperature readings. It also allows you to query various GPU attributes, such as clock speeds, power consumption, and supported features.

To monitor your GPU:

```

Nvidia-smi

```

To monitor your GPU in detail:

```

Nvidia-smi -q

```

To list all available NVIDIA devices:

```

Nvidia-smi -L

```

To list certain details about each GPU:

```

Nvidia-smi --query-gpu=index,name,uuid,serial --format=csv

```

To monitor overall GPU usage with 1-second update intervals:

```

Nvidia-smi dmon

```

To monitor GPU usage with process information:

```

Nvidia-smi pmon

```

Nvidia-smi output metrics and descriptions

The output of the nvidia-smi command includes two tables. The first table provides information about all available GPUs, while the second table shows the processes using the GPUs. Here are some key metrics and their descriptions:

• Temp: Core GPU temperature in degrees Celsius.
• Perf: Current performance state, ranging from P0 (maximum performance) to P12 (minimum performance).
• Persistence-M: Indicates whether the NVIDIA driver will remain loaded even when no active client is running.
• Pwr: Usage/Cap: GPU's current power usage out of total power capacity, sampled in Watts.
• Bus-Id: GPU's PCI bus ID in hex format.
• Disp.A: Display Active flag, indicating whether memory is allocated on the GPU for display initialisation.
• Memory-Usage: Memory allocation on the GPU out of total memory.
• Volatile Uncorr. ECC: Error count of ECC errors since the last driver load.
• GPU-Util: Percentage of GPU utilisation, indicating the percent of time when kernels were using the GPU.
• Compute M.: Compute Mode of the GPU, referring to the shared access mode. "Default" allows multiple clients to access the GPU simultaneously.
• GPU: GPU index, beneficial for multi-GPU setups, representing the NVML Index of the device.
• PID: Process ID of the process using the GPU.
• Type: Type of process, such as "C" (Compute), "G" (Graphics), or "C+G" (Compute and Graphics context).
• Process Name: Self-explanatory.
• GPU Memory Usage: Memory of a specific GPU utilised by each process.

For a full list of output metrics and descriptions, refer to the nvidia-smi manual page.

Using the watch command

The watch command is a useful tool for monitoring GPU usage in Linux, especially when used in conjunction with other commands such as nvidia-smi. By running watch -n 2 nvidia-smi --id=1, for example, you can automatically get updated measures of GPU utilisation and memory every two seconds. This is particularly helpful when you want to monitor your GPU for a few iterations of your code to understand its maximum GPU memory usage and utilisation.

The watch command can also be used with the nvidia-settings command for continuous GPU monitoring. For instance, watch -n0.1 "nvidia-settings -q GPUUtilization -q useddedicatedgpumemory" will provide real-time updates on GPU utilisation and dedicated GPU memory usage.

Additionally, the watch command can be combined with gpustat, a command-line script that wraps nvidia-smi, to monitor GPU status. For instance, watch -n 1 gpustat will provide continuous updates on GPU status without filling the terminal with output.

Using the nvtop command

Nvtop stands for Neat Video Card TOP, a (h)top-like task monitor for GPUs and accelerators. It can handle multiple GPUs and print information about them in a similar way to htop. Nvtop is supported by a range of vendors, including AMD, Apple, Huawei, Intel, NVIDIA, and Qualcomm.

To install nvtop, use the following command:

$ sudo apt install nvtop

For Ubuntu 20.04, 22.04, and newer:

$ sudo add-apt-repository ppa:flexiondotorg/nvtop

$ sudo apt install nvtop

To run the tool, simply type:

$ nvtop

While in nvtop, you can use the following commands:

• Up and Down arrows to select the previous or next process.
• Left and Right arrows to scroll in the process row.
• + to sort increasingly.
• - to sort decreasingly.
• F1 to select a signal to send to the highlighted process.
• F2 to select the field for sorting.
• F3, q, or Esc to exit nvtop and return to your shell.

For quick command-line arguments help, use:

$ nvtop --help

Nvtop on other operating systems

Ubuntu Impish (21.10) / Debian buster (stable) and more recent:

Install nvtop

$ sudo apt install nvtop

Fedora / Red Hat / CentOS:

Install nvtop

$ sudo dnf install nvtop

Red Hat Enterprise Linux 8 and 9:

$ sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(rpm -E %{rhel}).noarch.rpm

$ sudo dnf install nvtop

CentOS Stream, Rocky Linux, AlmaLinux:

$ sudo dnf install -y epel-release

$ sudo dnf install nvtop

OpenSUSE:

$ sudo zypper install nvtop

Arch Linux:

$ sudo pacman -S nvtop

Gentoo:

$ sudo layman -a guru && sudo emerge -av nvtop

AppImage:

Go to the release page and download nvtop-x86_64.AppImage

Go to the download location

$ cd $HOME/Downloads

Make the AppImage executable

$ chmod u+x nvtop-x86_64.AppImage

Enjoy nvtop

$ ./nvtop-x86_64.AppImage

Snap:

$ snap install nvtop

$ snap connect nvtop:process-control

$ snap connect nvtop:hardware-observe

$ snap connect nvtop:system-observe

$ snap connect nvtop:kubernetes-support

Conda-forge:

$ conda install --channel conda-forge nvtop

Pixi:

$ pixi global install nvtop

Docker:

$ git clone https://github.com/Syllo/nvtop.git && cd nvtop

$ sudo docker build --tag nvtop .

$ sudo docker run -it --rm --runtime=nvidia --gpus=all --pid=host nvtop

Build from source:

$ git clone https://github.com/Syllo/nvtop.git mkdir -p nvtop/build && cd nvtop/build

$ cmake .. -DNVIDIA_SUPPORT=ON -DAMDGPU_SUPPORT=ON -DINTEL_SUPPORT=ON

$ make

Install globally on the system

$ sudo make install

Alternatively, install without privileges at a location of your choosing

Make DESTDIR="/your/install/path" install

Using the intel-gpu-tools command

The intel-gpu-tools package comes with several commands, but the one we're interested in is intel_gpu_top, which provides a top-like overview of the current GPU usage.

To install intel-gpu-tools on Arch Linux, use the following command:

Sudo pacman -S intel-gpu-tools

Or a pacman wrapper of your choice, like yaourt.

If you're using another Linux-based operating system, you can download the source code and find packages for Debian, Fedora, and Ubuntu via the corresponding package managers.

Once installed, simply run the following command in your terminal:

Intel_gpu_top

This will display real-time GPU usage statistics for the Intel GPU, including the utilisation of various GPU components such as the GPU engine, memory, and the GPU core frequency.

To exit, simply press 'q'.

Note that you will need the appropriate drivers installed and your GPU must be supported by intel-gpu-tools for this to work.

Using the radeontop command

The `radeontop` command is a tool that can be used to monitor GPU usage on Linux systems. It displays the utilisation of AMD GPUs, including both the total activity percentage and individual block utilisation.

To install `radeontop`, use the following command:

Sudo apt install radeontop

Once installed, you can run the command by simply typing:

Radeontop

This command works with R600 and newer GPUs, including Southern Islands GPUs. It is compatible with both the open-source AMD drivers and the AMD Catalyst closed-source drivers.

$ sudo radeontop

GPU load: 100%

Memory clock: 150 MHz

Engine clock: 725 MHz

Memory usage: 2048 MB/8192 MB (25%)

GPU temperature: 71°C

In this example, the GPU load is at 100%, the memory clock is at 150 MHz, the engine clock is at 725 MHz, the memory usage is at 25%, and the GPU temperature is 71 degrees Celsius.

You can press `Ctrl+C` to exit the `radeontop` command and return to your shell prompt.

Frequently asked questions