Monitoring Xfs Performance: Tips For Optimizing Your System

XFS is a high-performance journaling file system created by Silicon Graphics, Inc. It is particularly good at parallel IO due to its allocation group-based design, which enables extreme scalability of IO threads, filesystem bandwidth, file and filesystem size when spanning multiple storage devices.

Performance analysis of XFS can be done with Performance Co-Pilot (PCP). PCP is installed and the XFS Performance Metrics Domain Agent (PMDA) is enabled by default. The Performance Metric Collector Daemon (PMCD) then begins collecting performance data from the installed Performance Metric Domain Agents (PMDAs). The XFS PMDA is used to gather performance metric data of XFS file systems in PCP.

PCP enables the reporting of certain XFS metrics per each of the mounted XFS file systems, making it easier to pinpoint specific mounted file system issues and evaluate performance. The pminfo command provides per-device XFS metrics for each mounted XFS file system.

Using the Performance Co-Pilot (PCP) to analyse XFS file system performance

Performance Co-Pilot (PCP) is a tool that can be used to analyse the performance of an XFS file system. The XFS PMDA, which is part of the default PCP installation, is used to gather performance metric data of XFS file systems in PCP.

Installing XFS PMDA to Gather XFS Data with PCP

The XFS PMDA ships as part of the pcp package and is enabled by default on installation. To install PCP, enter:

Systemctl enable pmcd.service

Systemctl start pmcd.service

To query the PCP environment to verify that the PMCD process is running on the host and that the XFS PMDA is listed as enabled in the configuration, enter:

Pcp Performance Co-Pilot configuration on workstation: platform: Linux workstation 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64 hardware: 2 cpus, 2 disks, 1 node, 2048MB RAM timezone: BST-1 services pmcd pmcd: Version 3.10.6-1, 7 agents pmda: root pmcd proc xfs linux mmv jbd2

Installing XFS PMDA Manually

If the XFS PMDA is not listed in PCP configuration readout, install the PMDA agent manually. The PMDA installation script prompts you to specify the PMDA role: collector, monitor, or both.

The collector role allows the collection of performance metrics on the current system, while the monitor role allows the system to monitor local systems, remote systems, or both. The default option is both collector and monitor, which allows the XFS PMDA to operate correctly in most scenarios.

To install XFS PMDA manually, change to the xfs directory:

Cd /var/lib/pcp/pmdas/xfs/

In the xfs directory, enter:

Xfs]# ./Install

Configuring and Examining XFS Performance Metrics

With PCP installed and the XFS PMDA enabled, the easiest way to start looking at the performance metrics available for PCP and XFS is to use the pminfo tool, which displays information about available performance metrics. The command displays a list of all available metrics provided by the XFS PMDA.

To display a list of all available metrics provided by the XFS PMDA:

Pminfo -t xfs.write_bytes xfs.write_bytes [number of bytes written in XFS file system write operations]

Pminfo -T xfs.read_bytes xfs.read_bytes Help: This is the number of bytes read via read(2) system calls to files in XFS file systems. It can be used in conjunction with the read_calls count to calculate the average size of the read operations to file in XFS file systems.

Pminfo -f xfs.read_bytes xfs.read_bytes value 4891346238

Configuring Metrics with pmstore

With PCP, you can modify the values of certain metrics, especially if the metric acts as a control variable, for example, the xfs.control.reset metric. To modify a metric value, use the pmstore tool.

Examining XFS Metrics Available per File System

Starting with Red Hat Enterprise Linux 7.3, PCP enables XFS PMDA to allow the reporting of certain XFS metrics per each of the mounted XFS file systems. This makes it easier to pinpoint specific mounted file system issues and evaluate performance.

Logging Performance Data with pmlogger

PCP allows you to log performance metric values that can be replayed later and used for a retrospective performance analysis. Use the pmlogger tool to create archived logs of selected metrics on the system.

With pmlogger, you can specify which metrics are recorded on the system and how often. The default pmlogger configuration file is /var/lib/pcp/config/pmlogger/config.default. The configuration file specifies which metrics are logged by the primary logging instance.

To log metric values on the local machine with pmlogger, start a primary logging instance:

Systemctl start pmlogger.service

Systemctl enable pmlogger.service

Visual Tracing with PCP Charts

To use the graphical PCP Charts application, install the pcp-gui package:

Yum install pcp-gui

You can use the PCP Charts application to plot performance metric values into graphs. The PCP Charts application allows multiple charts to be displayed simultaneously. The metrics are sourced from one or more live hosts with alternative options to use metric data from PCP log archives as a source of historical data. To launch PCP Charts from the command line, use the pmchart command.

Manually installing the PMDA agent

To manually install the PMDA agent, follow these steps:

Firstly, navigate to the xfs directory by entering the following command:

Cd /var/lib/pcp/pmdas/xfs/

Next, install the XFS PMDA manually. You will be prompted to choose an appropriate configuration and specify the PMDA role: collector, monitor, or both. The default option is both collector and monitor, which is suitable for most scenarios. Use the following command:

Xfs]# ./Install

You will see an output similar to the following:

You will need to choose an appropriate configuration for install of the “xfs” Performance Metrics Domain Agent (PMDA).

Collector collect performance statistics on this system

Monitor allow this system to monitor local and/or remote systems

Both collector and monitor configuration for this system

Please enter c(ollector) or m(onitor) or (both) [b]

Finally, verify that the pmcd process is running on the host and that the XFS PMDA is listed as enabled in the configuration. Enter the following command:

Pcp Performance Co-Pilot configuration on workstation

You should see an output similar to the following, indicating that the PMCD process is running and XFS PMDA is enabled:

Pcp Performance Co-Pilot configuration on workstation: platform: Linux workstation 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64 hardware: 12 cpus, 2 disks, 1 node, 36023MB RAM timezone: CEST-2 services: pmcd pmcd: Version 4.3.0-1, 8 agents pmda: root pmcd proc xfs linux mmv kvm jbd2

Examining XFS performance metrics with pminfo

Performance Co-Pilot (PCP) allows XFS PMDA to report certain XFS metrics for each of the mounted XFS file systems. This makes it easier to identify specific mounted file system issues and evaluate performance. The pminfo command provides per-device XFS metrics for each mounted XFS file system.

To display a list of all available metrics provided by the XFS PMDA, use the following command:

Pminfo --fetch --oneline xfs.perdev.read xfs.perdev.write

This will display the number of XFS file system read and write operations, along with the instance and value for each mounted XFS file system. For example:

Inst [0 or "loop1"] value 0

Inst [0 or "loop2"] value 0

You can also use the pminfo tool to display information about specific metrics. For example, to display a short description of the xfs.write_bytes metric, use the following command:

Pminfo --oneline xfs.write_bytes

This will output the following description:

[number of bytes written in XFS file system write operations]

To display a longer description of a metric, you can use the --helptext option. For example, to display a description of the xfs.read_bytes metric, use the following command:

Pminfo --helptext xfs.read_bytes

This will output the following description:

Help: This is the number of bytes read via read(2) system calls to files in XFS file systems. It can be used in conjunction with the read_calls count to calculate the average size of the read operations to file in XFS file systems.

You can also obtain the current performance value of a metric using the --fetch option. For example, to get the current value of the xfs.read_bytes metric, use the following command:

Pminfo --fetch xfs.read_bytes

This will output the metric name and its current value:

Xfs.read_bytes value 4891346238

In addition to the pminfo tool, there are other tools available in PCP for examining and configuring XFS performance metrics, such as pmstore and pmlogger.

Resetting XFS performance metrics with pmstore

Display the value of a metric:

Before making any changes, it is important to display the current value of the metric you plan to modify. This can be done using the pminfo command with the -f option, followed by the metric name. For example, to display the value of the xfs.write metric, you would use the following command:

$ pminfo -f xfs.write

Reset all the XFS metrics:

To reset all the XFS metrics, use the pmstore command followed by the metric name and the new value you want to set it to. In this case, we are resetting the xfs.control.reset metric to a value of 1. The command would look like this:

Pmstore xfs.control.reset 1

This will change the value of the metric from 0 to 1, as shown in the output:

Xfs.control.reset old value=0 new value=1

View the information after resetting the metric:

After resetting the metric, you can use the pminfo command again to view the updated information. This time, you can use the --fetch option to retrieve the current value of the metric. For example, to check the value of the xfs.write metric after resetting it, you would use the following command:

$ pminfo --fetch xfs.write

The output will show the updated value of the metric:

Xfs.write value 0

Additional resources:

For more detailed information about the pmstore and pminfo commands, you can refer to their respective man pages on your system. Additionally, you can explore the system services and tools distributed with PCP, which provide further insights into managing and monitoring XFS performance.

Using xfs_scrub to check online metadata for all XFS filesystems

To enable periodic checks of online metadata for all XFS filesystems, enable/start the `xfs_scrub_all.timer`. The timer runs every Sunday at 3:10 am and will be triggered immediately if it missed the last start time due to the system being powered off.

Note that `xfs_scrub` is an experimental program, meaning its behaviour and interface are subject to change. It is also an immature utility, so it is recommended that you do not run this program unless you have backups of your data.

`xfs_scrub` asks the kernel to scrub all metadata objects in the XFS filesystem. Metadata records are scanned for obviously bad values and then cross-referenced against other metadata. The goal is to establish a reasonable confidence about the consistency of the overall filesystem by examining the consistency of individual metadata records against the other metadata in the filesystem.

If redundant data structures are intact, damaged metadata can be rebuilt from other metadata. Filesystem corruption and optimisation opportunities will be logged to the standard error stream. Enabling verbose mode will increase the amount of status information sent to the output. If the kernel scrub reports that metadata needs repairs or optimisations, and the user does not pass `-n` on the command line, this program will ask the kernel to make the repairs and perform the optimisations.

The exit code returned by `xfs_scrub` is the sum of the following conditions:

• 0: No errors
• 1: File system errors left uncorrected
• 2: File system optimisations possible
• 4: Operational error
• 8: Usage or syntax error

Frequently asked questions