Monitoring Aix Process Performance: A Comprehensive Guide

Monitoring the performance of AIX processes is a crucial aspect of system administration. AIX, or Advanced Interactive eXecutive, is IBM's proprietary Unix operating system variant. It offers a range of tools and utilities for performance monitoring, including TOPAS for online monitoring and vmstat, iostat, and nmon for offline monitoring. The ps utility, for example, helps monitor process sizes, especially in cases of high CPU usage or performance issues. The Uptime Infrastructure Monitor AIX agent collects performance metrics using tools like sar, mpstat, ifconfig, and ps. SNMP (Simple Network Management Protocol) is another approach to monitoring AIX processes, although it requires additional configuration. IBM also provides Redbooks and forums with valuable information on this subject.

Using the ps utility to monitor process sizes

Monitoring the size of processes on AIX can be achieved using the ps utility. This is particularly useful in the case of 100% CPU usage or performance problems.

The following command can be used to monitor process sizes:

/ps.sh ps.log

This command uses a script that includes the following:

#!/bin/ksh PS_LOG=$1

Adjust LIMIT & SLEEP_TIME to suit your purpose/need.

There are 288 5-minute intervals in 1 day

LIMIT=288

Sleep for 5 minutes

SLEEP_TIME=300

While true

Do

I=0

Echo >$PS_LOG

While [ "$i" -le $LIMIT ]

Do

Date >>$PS_LOG

Ps avwwwg >>$PS_LOG

Echo >>$PS_LOG

I=`expr $i + 1`

Sleep $SLEEP_TIME

Done

Done

In this script, LIMIT refers to the number of log cycles before the log file rolls over, and SLEEP_TIME refers to the amount of time before each log cycle.

For example, to check the memory utilization of DataStage processes on AIX, the following command can be used:

Ps auxw | head -1; ps auxw | grep <process_name> | sort -r -n +5 | head -10

Here, `<process_name>` can be replaced with a specific DataStage process name, such as dsapi_slave, osh, or DSD.RUN. The output will list the top 10 processes based on memory utilization.

Monitoring CPU usage

On AIX systems, there are several methods and tools available for monitoring CPU usage:

Topas

The 'topas' command can be used to monitor CPU usage. Similar to the 'top' command in Linux, 'topas' provides a dynamic view of system processes and their CPU utilisation. By default, 'topas' displays a list of running processes along with their respective CPU usage percentages. The output can be sorted to easily identify the processes consuming the most CPU resources. For example, the command 'topas -P' displays a full screen of processes, allowing for a detailed view of CPU usage.

Ps

The 'ps' command is another tool available in AIX for monitoring CPU usage. It provides a snapshot of the current processes running on the system. By using the 'aux' option and sorting based on the relevant column, you can identify the processes with the highest CPU usage. For instance, the command 'ps aux | sort -nrk 3 | head -10' will display the top 10 processes with the highest CPU utilisation.

Sar

The 'sar' command is a powerful tool for monitoring system activity, including CPU usage. By using the 'u' option, followed by the desired interval and number of intervals, you can monitor CPU utilisation over a specific period. For example, 'sar -u 3 3' will execute 'sar -u' with a 3-second interval, three times consecutively.

Nmon and Topas

While 'nmon' is recommended as a comprehensive monitoring tool, 'topas' is also suggested as a useful alternative, albeit not as feature-rich as 'nmon'. These tools can provide insights into CPU usage and help identify any performance issues.

Perl Script

A Perl script has been developed for custom monitoring of CPU usage on IBM AIX servers. This script collects real CPU usage data, computational memory usage, paging rate, and disk space usage. It has been tested on AIX 6.3 and 7.1 versions and provides outputs such as Total CPU, Free CPU, Used CPU, and CPU Usage Ratio.

SNMP

Simple Network Management Protocol (SNMP) can be utilised to monitor CPU usage on AIX systems. While the native SNMP agent may not support the required MIB, using Net-SNMP can provide the necessary functionality. With the appropriate configuration, SNMP allows for remote monitoring of CPU utilisation and can integrate with other tools for comprehensive process monitoring.

Monitoring memory usage

Tools for Monitoring Memory Usage:

• Nmon: This tool provides a quick overview of memory usage by displaying some of the biggest memory consumers. After running nmon, pressing "m" will show you the computational memory usage, which includes both Process and System memory.
• Ipcs -m: This command lists the usage of UNIX Inter-process Communication (IPC), which can consume a significant amount of memory. It displays the shared memory used by applications such as DB2 and Oracle, and the Owner column typically indicates the purpose of the memory usage.
• Svmon: While nmon provides a simplified view, svmon gives the full details of memory usage, including all the memory attributes and libraries used by a process. It can be used to calculate the virtual memory used by each process and identify the highest memory-consuming processes.
• Vmstat: This command provides insights into virtual memory usage and free memory available on the LPAR. It helps determine if the system is over-committed in terms of memory and if adjustments are needed for improved performance.
• Topas: The output of topas includes the Computational percentage, which represents the Virtual memory usage. A low computational percentage indicates that the system is not memory-over-committed.
• Lsps -a: Examining paging space usage with the lsps -a command is another way to identify memory over-commitment. If the computational memory usage reaches 98% or more, the system starts paging out, and high paging space usage indicates the need for investigation.

Interpreting Memory Usage:

When monitoring memory usage, it's important to understand the different types of memory consumption:

• Computational Memory: This includes both Process and System memory usage, and it can be calculated by adding the Process and System percentages.
• Virtual Memory: Virtual memory usage is a measure of memory over-commitment. If the virtual memory reaches 98% or more of the total memory, the system is considered over-committed, and adjustments may be necessary.
• Free Memory: The fre column in vmstat indicates the amount of completely free memory, and if it's low compared to the total memory, increasing the allocated memory may improve performance.
• Pinned Memory: Pinned memory, which is non-computational, is not reflected in the AVM figure but can contribute to low free memory.

In summary, monitoring memory usage in AIX involves utilizing tools like nmon, svmon, vmstat, topas, and lsps -a to gain insights into memory consumption. By understanding the different types of memory usage and their interactions, administrators can make informed decisions to optimize system performance and address memory-related issues effectively.

Using the TOPAS command for online performance monitoring

The TOPAS command is used for online performance monitoring of AIX OS. When the command is run, a dynamic output is published onscreen, which changes continuously every 2 seconds. The information available from TOPAS includes kernel, user, wait, RAM, processes, CPU, NFS, and disk allocation.

The TOPAS command can be used to list all processes individually with the command "# topas –P". Similarly, disk information can be listed individually with "# topas –D". To quit the TOPAS output, use the "q" command.

The TOPAS command uses the curses library to display its output in a format suitable for an 80x25 character-based display or in a window of at least the same size on a graphical display. The output consists of two fixed parts and a variable section. The top seven lines display the name of the system, the date and time of the last observation, and the monitoring interval. The second fixed part contains five subsections of statistics, including system-global events, file and tty statistics, paging statistics, real memory size and distribution, and the size and utilization of paging space.

The variable part of the TOPAS display can include one, two, or three subsections: network interfaces, physical disks, and processes. The number of each can be specified using the -n, -d, and -p flags, respectively. If no flags are provided, TOPAS runs with the following default command line:

> topas -d5 -i2 -n2 -p16

The TOPAS command is a useful tool for monitoring the performance of AIX OS, providing detailed information about system activity and resource utilization.

Using the svmon command to capture and analyse snapshots of virtual memory

The svmon command is a powerful tool for capturing and analysing snapshots of virtual memory on AIX systems. It provides detailed insights into memory usage and can be used to identify and diagnose performance issues.

To get started with svmon, simply enter "svmon" at a command prompt as root. This basic form of the command will display the amount of memory on your system, broken down by page size. By default, svmon shows memory in 4K pages, but you can add the "unit" option to display in more readable megabytes or gigabytes.

One common use case for svmon is to identify processes that are consuming excessive memory. While the ps command provides basic process information, svmon goes further by showing how many pages of each size (small, medium, large, and huge) are being used and how much paging occurs for Small and Medium processes. This makes it easy to zero in on memory leaks and identify abnormal memory usage patterns.

Svmon also offers flexibility with its dozens of flags and arguments, which can be combined in numerous ways to customise your memory analysis. For example, you can run svmon with an interval rate and a number of data samples to capture memory usage over time, helping you detect peaks and troughs in memory consumption. This can be especially useful for troubleshooting intermittent performance issues.

Additionally, svmon provides options for different types of reports, including command reports, detailed segment reports, global reports, process reports, segment reports, and workload management reports. You can also choose to present svmon data in XML or ASCII text format, whichever suits your analysis needs best.

With its extensive capabilities and customisation options, svmon is an indispensable tool for system administrators managing AIX environments. By leveraging svmon's ability to capture and analyse snapshots of virtual memory, administrators can effectively monitor memory usage, identify issues, and optimise system performance.

Frequently asked questions