Monitoring Logstash Performance: Key Strategies For Success

Logstash is a powerful log aggregator, but it can be challenging to configure and debug. To monitor its performance, you can utilise tools like the Logstash Pipeline Viewer, which is part of the X-Pack monitoring features. This tool provides visual insights into your Logstash pipelines, allowing you to identify potential bottlenecks and optimise performance. Additionally, you can collect Logstash monitoring data using methods like Metricbeat, which sends data directly to your monitoring cluster, even if the Logstash instance is inactive. Other tools like the Hot Threads API can provide details on abnormal Java threads with high CPU usage, while the Elastic Stack monitoring features offer insights into the health of Logstash instances in your environment.

Utilise the Elastic Stack monitoring features to gain insight into the health of Logstash instances

To monitor Logstash performance, you can utilise the Elastic Stack monitoring features. This provides insight into the health of Logstash instances running in your environment. Before you begin, ensure that monitoring is enabled on your Elasticsearch cluster.

There are several methods to collect Logstash metrics:

Metricbeat Collection

Metricbeat collects data from your Logstash instance and sends it directly to your monitoring cluster. This method is advantageous as the monitoring agent remains active even if the Logstash instance is not.

Legacy Collection

Legacy collectors send monitoring data to your production cluster. However, this method is deprecated and may not offer the same level of features and dependability as other options.

Elastic Agent Collection

Elastic Agent collects data from your Logstash instance and sends it to your monitoring cluster, displaying the information in Logstash Dashboards. Like Metricbeat, the Elastic Agent's monitoring agent remains active even if the Logstash instance is not, and you can manage all your monitoring agents from a central location in Fleet.

Monitoring APIs

Introduced in version 5.0, Monitoring APIs provide general information and statistics on the Logstash node and installed plugins. The Hot Threads API, for example, offers details on abnormal Java threads with high CPU usage and long execution times.

Logstash Pipeline Viewer

The Logstash Pipeline Viewer is a monitoring feature offered in X-Pack. It provides a visual depiction of statistical data captured for the pipeline, allowing you to view data flow, branching logic, and plugin-specific information. This tool helps identify and resolve potential parsing bottlenecks, such as excessive CPU consumption by a grok filter.

Use Metricbeat collection to collect monitoring data from your Logstash instance

Metricbeat collection is a valuable method for collecting monitoring data from your Logstash instance. This approach offers the advantage of keeping the monitoring agent active, even if the Logstash instance is not. This ensures uninterrupted data collection and monitoring capabilities.

To effectively utilise Metricbeat collection, follow these comprehensive steps:

Firstly, ensure that you have Elasticsearch with Metricbeat monitoring set up. This is a prerequisite for collecting Logstash monitoring data with Metricbeat.

Next, disable the default collection of monitoring metrics. This can be achieved by setting the monitoring.enabled to false in the logstash.yml file.

Then, determine the target Elasticsearch cluster. This step involves specifying the cluster_uuid, which is necessary for binding the Logstash metrics to the correct cluster. If your pipeline includes Elasticsearch output plugins, the cluster_uuid is automatically calculated. However, if your pipeline lacks these plugins or you want to override automatic values, define the target cluster in the monitoring.cluster_uuid setting within the configuration file (logstash.yml).

Now, install and configure Metricbeat. Install Metricbeat on the same server as Logstash for optimal functionality. Enable the logstash-xpack module in Metricbeat to facilitate data collection from Logstash. You can enable the default configuration in the Metricbeat modules.d directory by running the command:

Metricbeat modules enable logstash-xpack

Configure the logstash-xpack module in Metricbeat by specifying the required settings in the modules.d/logstash-xpack.yml file. These settings include module, metricsets, period, hosts, username, and password. The default collection host is localhost:9600, but you can customise this as needed.

If you intend to monitor multiple Logstash instances, provide a list of hosts in the hosts setting. For instance:

Hosts: ["http://localhost:9601","http://localhost:9602","http://localhost:9603"]

Additionally, if you have enabled Elastic security features, create a user on the production cluster with the remote_monitoring_collector built-in role. Alternatively, you can use the remote_monitoring_user built-in user if it is available. Add the username and password settings to the module configuration file (logstash-xpack.yml) for authentication.

Optionally, you can disable the system module in Metricbeat. While the system module is enabled by default, the information it collects is not displayed on the Stack Monitoring page in Kibana. To disable it, run the following command:

Metricbeat modules disable system

Finally, identify where to send the monitoring data. In production environments, it is highly recommended to use a separate cluster, known as the monitoring cluster, to store the data. This practice ensures that any issues with the production cluster do not impede your access to monitoring data and prevents monitoring activities from affecting the performance of the production cluster.

Specify the Elasticsearch output information in the Metricbeat configuration file (metricbeat.yml). Here is an example of how to structure this information:

Output.elasticsearch:

Hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"]

If you have configured the monitoring cluster for encrypted communications, remember to access it via HTTPS.

By following these steps, you can effectively use Metricbeat collection to gather monitoring data from your Logstash instance, benefiting from continuous monitoring capabilities and gaining valuable insights into the health and performance of your Logstash environment.

Legacy collection (deprecated) sends monitoring data to your production cluster

Legacy collection (deprecated) is one of the methods to collect Logstash metrics. It involves using legacy collectors to send monitoring data to your production cluster. This is in contrast to Metricbeat collection, where data is sent directly to the monitoring cluster, and the monitoring agent remains active even if the Logstash instance is not.

The Elastic Stack monitoring features can be used to monitor the health of Logstash instances in your environment. To do this, ensure that monitoring is enabled on your Elasticsearch cluster. Then, you can configure one of the collection methods mentioned above to gather Logstash metrics.

While the Legacy collection method is still functional, it is considered deprecated, meaning it is no longer the recommended approach and may not receive future updates or support. This suggests that users should transition to more recent collection methods, such as Metricbeat or Elastic Agent collection, which offer improved features, dependability, and easier management.

For instance, Elastic Agent collection collects monitoring data from Logstash and sends it to the monitoring cluster, where it can be visualised in Logstash Dashboards. Additionally, the monitoring agent remains active even if the Logstash instance is inactive, and all monitoring agents can be managed centrally via Fleet.

Monitoring APIs provide general info and stats on the Logstash node and installed plugins

Monitoring Logstash with APIs allows users to retrieve runtime metrics about Logstash. Monitoring APIs were introduced in version 5.0 and can provide general information and statistics on the Logstash node and installed plugins.

The Node Info API can be used to get information about the nodes of Logstash, including the OS, Logstash pipeline, and JVM in JSON format. The following GET request can be sent to Logstash to extract this information:

GET http://localhost:9600/_node?pretty

The response will include details such as the host, version, http_address, pipeline settings, OS info, and JVM info.

To get specific information about the pipeline, OS, or JVM, you can add their names to the URL:

GET http://localhost:9600/_node/os?pretty

GET http://localhost:9600/_node/pipeline?pretty

GET http://localhost:9600/_node/jvm?pretty

Additionally, the Plugins Info API can be used to retrieve information about the installed plugins in Logstash. This can be achieved by sending a GET request to the following URL:

GET http://localhost:9600/_node/plugins?pretty

The response will include details such as the host, version, http_address, and a list of installed plugins with their names and versions.

Furthermore, Logstash provides APIs to extract statistics about its performance, including memory, process, JVM, and pipeline information. These statistics are returned in JSON objects and can be retrieved by sending GET requests to the following URLs:

GET http://localhost:9600/_node/stats/?pretty

GET http://localhost:9600/_node/stats/process?pretty

GET http://localhost:9600/_node/stats/jvm?pretty

GET http://localhost:9600/_node/stats/pipeline?pretty

Finally, the Hot Threads API allows users to get details on abnormal Java threads with high CPU usage and long execution times. This information can be retrieved by sending a GET request to the following URL:

GET http://localhost:9600/_node/hot_threads?pretty

The Logstash Pipeline Viewer is a tool for improving performance

Logstash is a powerful log aggregator, but it can be challenging to configure and debug. The Logstash Pipeline Viewer is a valuable tool for improving performance and monitoring Logstash pipelines. It was introduced in version 6.0 to help users visualise and understand their pipelines better.

The Pipeline Viewer is part of the monitoring features offered in X-Pack. It provides a visual representation of the statistical data captured for a pipeline, including data flow, branching logic, and plugin performance. Each component in the viewer has labels and metrics, such as worker usage, performance, and throughput, which can help identify potential issues. For example, high CPU usage by a particular plugin may indicate a future bottleneck.

The Pipeline Viewer UI offers additional visibility into the behaviour and performance of complex pipeline configurations. It displays a tree view illustrating the pipeline topology, data flow, and branching logic. The UI highlights anomalous values, such as high CPU% and event latency, helping users quickly identify slow processing. It also provides the ability to interact with the tree view, allowing users to click on elements like plugin names to expand the detail view and gain further insights.

The Pipeline Viewer is an excellent tool for identifying and resolving potential parsing bottlenecks. It allows users to compare the performance of different plugins and identify areas where alternative filters or optimisations can be implemented to improve overall pipeline performance.

Frequently asked questions