Monitoring Pci Bus Usage: Linux Command Line Essentials

Linux users can monitor their PCI bus usage through the lspci command, which displays information about all the PCI and PCIe buses in a server, as well as the hardware devices connected to them. This includes Ethernet cards, RAID controllers, and video cards. The lspci utility is part of the pciutils package, which can be installed using the yum command.

Other tools for monitoring PCI bus usage include the Intel Performance Counter Monitor (PCM) and the Nvidia NVAPI programming interface.

Using the lspci command to get PCI bus hardware device info

The lspci command is a useful tool for listing and identifying PCI devices on your Linux system. It is available on every Debian system and can be used to display information about all the PCI buses and their attached devices.

To use lspci, simply enter the following command:

Lspci

This will provide you with a list of all the PCI buses and their corresponding devices. Each device will be listed with details such as its slot information, vendor, name, and model number. For example:

00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)

00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)

00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)

You can also use the lspci command to look up specific devices by slot number or vendor/device ID. For example:

Lspci -s 03:00.0

Lspci -d 1000:0079

Additionally, the lspci command can be used to display kernel driver modules and their associated devices:

Lspci -k

By using the lspci command, you can easily identify and gather detailed information about the PCI devices connected to your Linux system.

Determining the bus address of the device in question

To determine the bus address of a device in Linux, you can use the lspci command. This will list all the PCI devices connected to your system, along with their respective bus addresses.

For example, to find the PCI bus number of an Ethernet interface, you can use the following command:

Bash

Lspci -D | grep 'Network\|Ethernet'

This will filter the output to only show the Ethernet controller and its corresponding bus address.

Another option is to use the lshw command, which also displays information about the PCI devices on your system:

Bash

Lshw -class network -businfo

This will show the network devices and their respective bus addresses.

Additionally, you can use the dmidecode command to get information about the PCI slots on your system:

Bash

Dmidecode -t slot

This will output the current usage and bus address of each slot.

Once you have the bus address of the device in question, you can use it to get more detailed information about the device. For example, you can use the lspci -nv command to see the negotiated speed of the device:

Bash

Lspci -nv

This will display the "LnkCap" and "LnkSta" information, which includes the negotiated speed of the device.

Alternatively, you can use the dmidecode command to search for the bus address and get information about the slot type and length:

Bash

Dmidecode

This will display the "Designation" and "Type" fields, which include the slot type and length.

Using the Intel Performance Counter Monitor (PCM) tools

The Intel Performance Counter Monitor (PCM) is an application programming interface (API) and a set of tools that can be used to monitor performance and energy metrics of Intel Core, Xeon, Atom, and Xeon Phi processors. It supports various operating systems, including Linux, Windows, and Mac OS X.

PCM provides a range of command-line utilities for real-time monitoring, such as:

• Pcm: A basic processor monitoring utility that measures instructions per cycle, core frequency, memory and Intel Quick Path Interconnect bandwidth, cache misses, core and CPU package sleep C-state residency, cache utilisation, and CPU and memory energy consumption.
• Pcm-sensor-server: A collector that exposes metrics over HTTP in JSON or Prometheus format.
• Pcm-memory: A tool to monitor memory bandwidth per-channel and per-DRAM DIMM rank.
• Pcm-accel: A utility to monitor Intel In-Memory Analytics Accelerator, Intel Data Streaming Accelerator, and Intel QuickAssist Technology accelerators.
• Pcm-latency: A tool to monitor L1 cache miss and DDR/PMM memory latency.
• Pcm-pcie: Monitor PCIe bandwidth per-socket.
• Pcm-iio: Monitor PCIe bandwidth per PCIe device.
• Pcm-numa: Monitor local and remote memory accesses.
• Pcm-power: Monitor sleep and energy states of the processor, Intel Quick Path Interconnect, DRAM memory, reasons for CPU frequency throttling, and other energy-related metrics.
• Pcm-tsx: Monitor performance metrics for Intel Transactional Synchronization Extensions.
• Pcm-core and pmu-query: Query and monitor arbitrary processor core events.
• Pcm-raw: Program arbitrary core and uncore events by specifying raw register event ID encoding.
• Pcm-bw-histogram: Collect memory bandwidth utilisation histogram.

In addition to these command-line tools, PCM also offers graphical front-ends such as the pcm Grafana dashboard and pcm-sensor for KDE KSysGuard and Windows perfmon.

To use PCM, you can either clone the GitHub repository and build it from source or download pre-compiled binaries for your specific operating system. For Linux, the PCM repository can be cloned with submodules using the following command:

> git clone --recursive https://github.com/intel/pcm

Alternatively, you can first clone the repository and then update submodules with:

> git submodule update --init --recursive

Once you have cloned the repository, you can install CMake (and libasan on Linux) and build the project using the following commands:

> mkdir build

> cd build

> cmake ..

> cmake --build .

The built utilities will be located in the build/bin directory. For faster building, you can use the '--parallel' option:

> cmake --build . --parallel

On Windows and macOS, additional drivers and steps are required, which are outlined in the respective HOWTO files in the repository.

Using the bandwidthtest.cu code from Nvidia

The bandwidthtest.cu code from Nvidia can be used to measure the data transfer rate between the GPU and the CPU. The code is available on Nvidia's developer blog and can be compiled using the following commands:

Git clone https://github.com/parallel-forall/code-samples.git

Cd code-samples/series/cuda-cpp/optimize-data-transfers

Nvcc bandwidthtest.cu -o bandwidthtest

Nvprof ./bandwidthtest

The code includes functions to check CUDA runtime API results and perform CUDA memory copies and allocations. It measures the bandwidth for device-to-host and host-to-device transfers for both pageable and pinned memory. The output includes the device name, transfer size, and bandwidth for each type of transfer.

Device: GeForce GTX TITAN X

Transfer size (MB): 3960

Pageable transfers

Host to Device bandwidth (GB/s): 3.073613

Device to Host bandwidth (GB/s): 3.588289

Pinned transfers

Host to Device bandwidth (GB/s): 12.004806

Device to Host bandwidth (GB/s): 12.929138

The code also includes options to specify the memory mode (pageable or pinned), the device to be used, and the mode of operation (quick, range, or shmoo). The quick mode performs a single measurement, the range mode measures a user-specified range of values, and the shmoo mode performs an intense shmoo of a large range of values.

The bandwidthtest.cu code can be useful for optimizing data transfers between the GPU and CPU and identifying potential bottlenecks in the PCI bus.

Using the pcitop tool

The PCITOP package is a tool used to monitor PCI bus utilization. It does this by poking and prodding counters in the root bridge chipsets found in a system, parsing the data, and producing a display showing utilization as a percentage for each root bridge.

There are two components to the pcitop project: a driver called hplba, and a user tool called pcitop. The driver provides the pci utilization counters that the user land tool queries. The driver exposes the counters using the sysfs file-system. The user tool reads and writes these sysfs entries to obtain a count of elapsed time and the time that the pci bus was in use (reading or writing data). It then calculates a ratio of these values to get utilization.

The pcitop tool then displays snapshots of utilization for each PCI bridge in the system. The tool uses the acpiphp or pci_slot driver to map PCI slots to root bridges.

Pcitop operates in two modes. The first mode is used to present information about root bridges found in the system. Here is an example:

Pcitop -i

0000:4a:00.0 PCI/66 MHz lba id = 0x122e rev=3.2 in use=n ropes=1 cabinet 0 bay 0 chassis 0 slots 8

0000:49:00.0 PCI-X mode 1/66 MHz lba id = 0x122e rev=3.2 in use=y ropes=1 cabinet n/a bay n/a chassis n/a slots n/a

0000:0a:00.0 PCI/66 MHz lba id = 0x122e rev=3.2 in use=n ropes=2 cabinet 0 bay 0 chassis 0 slots 7

0000:01:00.0 PCI-X mode 1/66 MHz lba id = 0x122e rev=3.2 in use=y ropes=1 cabinet 0 bay 0 chassis 0 slots 10,9

0000:00:00.0 PCI/66 MHz lba id = 0x122e rev=3.2 in use=y ropes=1 cabinet n/a bay n/a chassis n/a slots n/a

0000:c4:00.0 PCIe/x8 lba id = 0x4037 rev=2.2 in use=y ropes=4 cabinet n/a bay n/a chassis n/a slots n/a

0000:4f:00.0 PCIe/x8 lba id = 0x4037 rev=2.2 in use=y ropes=2 cabinet n/a bay n/a chassis n/a slots n/a

0000:0f:00.0 PCIe/x8 lba id = 0x4037 rev=2.2 in use=y ropes=4 cabinet n/a bay n/a chassis n/a slots n/a

The second mode is used to display bus utilization. It measures utilization over a given interval and outputs the utilization for each root bridge. Various filters can be applied to limit the number of root bridges displayed. Example:

Pcitop

Frequently asked questions