Monitoring Apache Web Server Performance: A Comprehensive Guide

Apache is an open-source web server that is widely used for hosting websites. It is important to monitor Apache server performance to maintain robust and efficient server functionality. The performance metrics can be accessed through the Apache server status module (mod_status) and server access log. Monitoring Apache web server performance helps to identify potential problems and ensure a quality, easy-to-scale user experience.

Monitoring the rate of requests, request latency, and error rates

Understanding Metrics and Tools:

• Throughput and Latency Metrics: These include the request processing time (in microseconds) and the average number of client requests per second. Monitoring these metrics can help identify slowdowns caused by factors such as long-running queries or inefficient code.
• Resource Utilization and Activity Metrics: This category includes Multi-Processing Modules (MPMs) that determine how Apache uses system resources to handle requests. Understanding MPMs helps optimise resource utilisation and identify bottlenecks.
• Host-level Resource Metrics: Monitoring host-level resources such as memory usage, CPU utilisation, and open file descriptors provides a comprehensive view of server performance and helps identify scaling needs.
• Apache's Status Module (mod_status): Apache exposes high-level metrics through mod_status, providing real-time information about server performance. It requires enabling and proper configuration to access the status page and relevant metrics.
• Apache's Access Log: The access log provides detailed information about each client request, including request time, client IP, HTTP status codes, and response size. Customising log formats and using log aggregation tools can aid in performance analysis.

Best Practices and Recommendations:

• Monitor Rate of Requests: Keep track of sudden increases or decreases in the rate of requests. A dramatic increase may require additional resources or indicate illegitimate traffic like a denial-of-service attack. A rapid decrease could point to issues such as server swapping to disk or database problems.
• Optimise Request Latency: To improve request latency, consider using NGINX as a proxy for static content, refer to Apache's caching guide, or use a cache system like Varnish. Additionally, ensure that HostnameLookups are disabled and access restrictions are based on IP addresses rather than hostnames for better performance.
• Analyse Error Rates: Monitor both client errors (e.g., 404 Not Found) and server errors (e.g., 500 Internal Server Error). A high rate of client errors may indicate potential security issues or outdated links. Server errors may point to configuration issues or resource limitations.
• Optimise Resource Utilisation:
• Memory Usage: Monitor memory usage to prevent Apache from running out of memory and degrading performance. Consider switching from prefork to worker or event MPM, disabling unnecessary Apache modules, or scaling horizontally by adding more servers.
• CPU Utilisation: Continuously rising CPU usage indicates insufficient resources to handle the request load. Consider moving database and application servers to separate machines to improve scalability.
• Open File Descriptors: Keep an eye on the number of open file descriptors, especially in servers with many virtual hosts, to avoid reaching system-imposed limits.

Tracking server load, uptime, and total accesses

Apache mod_status:

The Apache module, mod_status, provides valuable insights into the server's performance. This module is usually enabled by default and offers real-time information about web server load, current connections, total requests received, server load and uptime, active vs. idle workers, and process IDs of connected clients.

To access the mod_status page, go to: http://192.0.2.0/server-status. If this link doesn't work, you may need to enable mod_status in your configuration file. The specific steps depend on your operating system:

• Debian systems: Edit the status module's configuration file at /etc/apache2/mods-enabled/status.conf.
• Other UNIX-like platforms: Edit the main Apache configuration file at /etc/apache2/apache2.conf, /etc/httpd/conf/httpd.conf, or /etc/apache2/httpd.conf. Ensure the line "LoadModule status_module libexec/apache2/mod_status.so" is uncommented.

After locating and enabling mod_status, you may need to restart Apache for the changes to take effect.

ExtendedStatus:

To get more detailed metrics like CPU load and requests per second, enable ExtendedStatus in your Apache configuration file. This option provides additional information about each request, including client processing time and the requested source. It also offers metrics such as:

• ServerVersion: Apache version information.
• ServerMPM: Multi-Processing Module in use.
• ServerUptimeSeconds: Uptime of the server in seconds.
• Total Accesses: Total number of requests received.
• CPUUser, CPUSystem: CPU usage metrics.
• ReqPerSec, BytesPerSec: Request and byte throughput metrics.

Access Logs:

In addition to mod_status, Apache's access logs provide valuable insights. These logs include details about each client request, such as the remote hostname, remote logname, time of the request, HTTP request method, endpoint/resource, and response status code.

You can customize the access logs by editing the LogFormat directive in the Apache configuration file. This allows you to specify the order and content of the logged information.

Open-source Tools:

There are also open-source tools available to help monitor Apache performance:

• Apachetop: Parses Apache access logs and displays statistics in the command line, similar to the "top" command. It provides metrics like the rate of requests, bytes served per second, and the percentage of requests resulting in different status codes.
• Atop: Retrieves information from mod_status (with ExtendedStatus enabled) and offers interactive ways to search and filter through requests.

Commercial Tools:

Commercial tools like Datadog offer Apache integration to visualise metrics and set up alerts. This helps to correlate performance data across different components of your stack.

Analysing logs to identify performance problems, error statuses, and attacks

Analysing logs is an important part of administering Apache and ensuring it runs as expected. There are two main types of Apache logs: access logs and error logs.

Access Logs

Access logs contain all server requests and provide details about who sends requests to the server and what data they request. Each request log contains details such as the client's IP address, timestamp, the requested URL, HTTP response code, and response body size.

The primary use of access logs is to analyse website traffic, discover potential security threats, and optimise web server performance. Monitoring access logs can help identify traffic patterns and inform decisions about server security and optimisation.

> 127.0.0.1 - - [09/Feb/2024:15:36:14 +0100] "GET / HTTP/1.1" 200 3460 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0"

In this entry, the client with the IP address 127.0.0.1 made a request to access the root URL ("/"), which resulted in a successful HTTP status of 200, and a response size of 3460 bytes. The referrer field is unavailable ("-") and the user agent is "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0".

Error Logs

Error logs contain critical error reports and details about the issues encountered by the server. They are used for troubleshooting and monitoring server problems, helping administrators identify and resolve issues quickly to ensure minimal downtime.

> [Fri Feb 09 15:35:24.252107 2024] [core:notice] [pid 6672:tid 139657266624384] AH00094: Command line: '/usr/sbin/apache2'

This error log entry includes the timestamp when the record was added ("Fri Feb 09 15:35:24.252107 2024"), the component that generated the log entry ("core") and its severity level ("notice"), the process and thread IDs ("pid 6672:tid 139657266624384"), and the error message ("Command line: '/usr/sbin/apache2').

Identifying Performance Problems and Error Statuses

By examining the access and error logs, administrators can gain insights into the performance and health of the Apache web server. For example, the access log can show the rate of requests and the size of responses, while the error log can indicate issues with missing resources or internal server errors.

The HTTP status codes in the access log can be particularly informative, with codes in the 2xx range indicating successful requests, 3xx indicating redirections, 4xx indicating client errors, and 5xx indicating server errors.

Additionally, the error log can provide details about missing resources or potential server problems, such as a requested page that was not found (404 error).

Identifying Attacks

Analysing Apache logs can also help detect potential security threats and identify suspicious activity. For example, by examining access logs, administrators can look for patterns such as multiple requests from the same IP address in a short time frame, repeated access to secure or login pages, or attempts to access non-existent pages using different query parameters.

Third-party tools and modules can also assist in detecting attacks by parsing log files and identifying specific patterns or anomalies. For example, tools like apache-scalp, mod_sec, and WebForensik can help analyse logs for signs of web app attacks.

In summary, by effectively monitoring and analysing Apache access and error logs, administrators can identify performance problems, error statuses, and potential security threats, ensuring the smooth operation and security of the web server.

Monitoring memory usage and CPU utilisation

Memory Usage

High memory usage can cause a server to crash. To avoid this, you can tweak the settings of the Multi-Processing Module (MPM) of Apache. By default, most Apache software comes with the Prefork module, but for high-traffic servers, the Worker MPM is a better option. The Prefork module uses multiple child processes, with each child handling one connection at a time, which leads to high memory usage. On the other hand, the Worker MPM uses multiple child processes with many threads each, and each thread handles one connection at a time, resulting in a lower memory footprint.

You can also minimise simultaneous connections by tweaking parameters such as MaxClients, which sets the limit on the number of simultaneous requests that can be supported by the Apache web server. KeepAlive is another parameter that can be adjusted; it keeps a single connection alive to transfer all the files to load a page, but these connections occupy RAM that could be used for other services. Therefore, it is recommended to set the value of KeepAliveTimeout to 2-5 seconds for low-traffic servers and 10 seconds for high-traffic servers.

Another way to reduce memory usage is to disable any unnecessary Apache modules. Apache loads all enabled modules into the server memory, which can slow down the server. For example, if you are using CentOS 6.x, you can disable any unused modules such as proxy_balancer_module, proxy_ftp_module, proxy_http_module, proxy_ajp_module, and so on.

CPU Utilisation

Constant high CPU usage indicates a shortage of resources to handle the current requests. If you are running a database or application server on the same host as Apache, consider moving them to separate machines to give you more flexibility to scale each layer of your environment.

If you are using a multi-core system, you should monitor CPU utilisation across all cores. Additionally, keep an eye on the number of open file descriptors, as Apache opens a file descriptor for each connection and log file. If your server has many virtual hosts, you may encounter problems with your system-imposed limit.

To summarise, monitoring memory usage and CPU utilisation is crucial for maintaining the performance and stability of your Apache web server. By adjusting the settings of the Multi-Processing Module, minimising simultaneous connections, disabling unnecessary modules, and keeping track of CPU usage across all cores and open file descriptors, you can optimise your server's resource utilisation and ensure it runs efficiently.

Using tools like Datadog, ManageEngine Applications Manager, and Paessler PRTG Network Monitor

There are several tools available to monitor Apache web server performance. Here's an overview of how three popular tools—Datadog, ManageEngine Applications Manager, and Paessler PRTG Network Monitor—can help with this task:

Datadog

Datadog is a powerful monitoring platform that offers a wide range of features, including infrastructure monitoring, application performance monitoring, and log management. With Datadog, you can monitor Apache web server performance by:

• Setting up the Apache integration: Datadog's Apache integration allows you to automatically collect various Apache metrics, such as request processing time, throughput, and error rates.
• Collecting and monitoring Apache access logs: Datadog can parse and analyze Apache access logs, providing insights into request details, HTTP response codes, and more.
• Enabling automated alerts: Datadog allows you to set up alerts to notify you of performance issues in real time, such as when an Apache server goes down or when specific metrics cross predefined thresholds.

ManageEngine Applications Manager

ManageEngine Applications Manager is a comprehensive solution for monitoring Apache web server performance. It offers the following capabilities:

• Real-time performance insights: It provides insights into critical performance metrics, such as Apache response time, incoming requests, worker resource metrics, server connections, and system load.
• Issue identification and troubleshooting: The tool helps identify performance issues and provides proactive alerts, enabling administrators to troubleshoot and resolve problems quickly.
• Customizable alerts and baselines: You can configure alarms and threshold profiles for different performance attributes, receive notifications for threshold violations, and set up dynamic baselines to detect anomalies.
• Performance analysis and forecasting: Applications Manager offers historical trend analysis and machine learning-based forecasting of server performance, helping administrators plan resource allocation and capacity management.

Paessler PRTG Network Monitor

PRTG Network Monitor is an all-in-one web server monitoring solution that supports various web servers, including Apache. Here's how it can help monitor Apache web server performance:

• Availability and performance monitoring: PRTG monitors the availability and performance of your Apache web servers, tracking parameters such as CPU load, memory usage, web server performance, and website load times.
• Customizable sensors: PRTG offers preconfigured and custom sensors for monitoring Apache web servers. These sensors track various aspects, including availability, requests, transferred data, uptime, and CPU load.
• Visualisation and notification: PRTG provides visual dashboards and notifications to help you easily track critical monitoring data. You can set up alerts to receive immediate notifications when issues occur, such as server unavailability or increased queries and load.
• Compatibility and flexibility: PRTG is compatible with major web server manufacturers and can be set up as a stationary or cloud-based solution, providing flexibility for different deployment needs.

Frequently asked questions