Monitoring Memory Usage: Emr Spark Management Strategies

Monitoring memory usage in EMR Spark is crucial for optimising performance and resource allocation. While the built-in EMR cluster monitoring tool Ganglia provides an overview of CPU and memory utilisation, it may not offer detailed insights into memory usage by individual executors. To delve deeper, one can refer to the Spark UI's storage and executor tabs or leverage external tools like VisualVM for local development. Additionally, configuring spark metrics properties and utilising EMR's special setting maximiseResourceAllocation can aid in maximising memory and CPU allocation. However, this setting may not be suitable for network-bound or CPU-bound jobs, where multiple containers or smaller instances, respectively, could be more beneficial.

Utilising the Executor tab in Spark UI

The Executors tab in Spark UI provides a summary of the executors created for a Spark application, including memory and disk usage, task information, and shuffle data. This tab is useful for monitoring resource usage and performance metrics related to executors.

To access the Executors tab, you can use the URL http://localhost:4040/executors. This tab provides detailed insights into the memory and disk usage of each executor, helping you understand how your application is utilising system resources.

The Executors tab offers valuable information such as the number of cores used by each executor, allowing you to make informed decisions about resource allocation and optimisation. Additionally, you can monitor the GC (Garbage Collection) time, which indicates the time spent on memory management tasks, and shuffle information, which includes data movement across executors.

By utilising the Executors tab, you can identify potential bottlenecks or resource contention issues related to memory usage. For example, if you observe high memory usage across multiple executors, you may need to adjust your memory settings or distribute your data across a larger number of executors.

Furthermore, the Executors tab provides performance insights beyond memory usage. For instance, you can analyse the "Thread Dump" feature, which offers a thread dump of the JVM on a specific executor, aiding in performance analysis and troubleshooting.

Using the RM UI to display total memory per application

To use the RM UI to display total memory per application, follow these steps:

Firstly, it is important to note that the RM UI is the Resource Manager User Interface, and it displays the total memory consumption of a Spark app that has executors and drivers.

To access the Spark Web UI, which includes the RM UI, you can do so in two ways:

• Spark Driver Web UI: This is the most comprehensive interface as it provides live data on your applications in terms of disk and memory utilisation. You can access this by connecting to the YARN Resource Manager and then opening the Application Master URL for the running Spark application.
• Spark History Server: This service allows you to review completed or running applications on the cluster. It is available on the EMR Primary (MASTER) node and can be reached at http://master-public-dns-name:18080/. Note that this interface uses Spark Event Logs, so information only available at runtime may not be present.

Once you have accessed the Spark Web UI, you can navigate to the RM UI to view the total memory per application. This will display the total memory consumption of the Spark app.

The RM UI provides valuable insights into the memory utilisation of your Spark applications, helping you to optimise and troubleshoot your Spark workloads.

Configuring spark metrics properties

To configure spark metrics properties, you can use the metrics configuration file or configuration parameters with the prefix "spark.metrics.conf." followed by the configuration details. The default metrics configuration file is "metrics.properties" and is expected to be present at $SPARK_HOME/conf/metrics.properties. A custom file location can be specified via the "spark.metrics.conf" configuration property.

The configuration file defines the syntax as:

> [instance].sink|source.[name].[options]=[value]

Where:

• [instance] refers to the component instance, such as "master", "worker", "executor", "driver", or "applications".
• [name] specifies the name of the source or sink.
• [options] refers to the specific property of the source or sink.

For example, to enable the JmxSink for all instances, the configuration would be:

> *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink

To enable the ConsoleSink for all instances, the configuration would be:

> *.sink.console.class=org.apache.spark.metrics.sink.ConsoleSink

You can also specify polling periods for certain sinks. For example, to set the polling period for the ConsoleSink to 10, the configuration would be:

> *.sink.console.period=10

Additionally, you can override wildcard properties with more specific properties. For instance, to set a polling period for the ConsoleSink specific to the master instance, you can use:

> master.sink.console.period=15

Spark also provides a list of available common sources and their properties, such as the JvmSource, which collects low-level state.

Furthermore, Spark supports various sinks, including:

• ConsoleSink: Logs metrics information to the console.
• CSVSink: Exports metrics data to CSV files at regular intervals.
• JmxSink: Registers metrics for viewing in a JMX console.
• MetricsServlet: Adds a servlet within the Spark UI to serve metrics data as JSON.
• PrometheusServlet: Adds a servlet within the Spark UI to serve metrics data in Prometheus format.
• GraphiteSink: Sends metrics to a Graphite node.
• Slf4jSink: Sends metrics to slf4j as log entries.
• StatsdSink: Sends metrics to a StatsD node.

To monitor the memory usage of a Spark application, you can refer to the list of available metrics providers. For example, under the "Component instance = Driver" section, you can find metrics such as "memory.maxOffHeapMem_MB", "memory.maxOnHeapMem_MB", and "memory.onHeapMemUsed_MB".

Using the spark-ui storage and executor tabs

The Executors tab in the Spark Web UI provides summary information about the executors created for the application, including memory and disk usage, task and shuffle information, and performance information. The Storage Memory column shows the amount of memory used and reserved for caching data.

The Executors tab is useful for monitoring resource information, such as the amount of memory, disk, and cores used by each executor. It also provides performance insights, including GC time and shuffle information. For example, clicking on the 'stderr' link of an executor displays its detailed standard error log. Similarly, clicking on the 'Thread Dump' link provides the thread dump of the JVM on that executor, which is valuable for performance analysis.

The Executors tab also displays the number of cores and tasks. In a scenario where the Spark application was run locally with master set to local with three threads, the Executors tab showed three cores and four tasks.

The Storage tab in the Spark Web UI displays the persisted RDDs and DataFrames in the application. The summary page shows storage levels, sizes, and partitions of all RDDs, while the details page presents the sizes and executors for all partitions in an RDD or DataFrame.

Using the Ganglia dashboard

The Ganglia dashboard is a powerful tool for monitoring memory usage in EMR Spark applications. Ganglia is a scalable, distributed system designed to monitor clusters and grids while minimising performance impact.

When you open the Ganglia web UI in a browser, you are presented with an overview of the cluster's performance, including detailed graphs showing load, memory usage, CPU utilisation, and network traffic. This high-level view provides a comprehensive snapshot of the cluster's health and can quickly identify any performance bottlenecks or issues.

Scrolling down the dashboard, you will find individual server graphs, allowing you to inspect the performance of each node in the cluster. This level of granularity is invaluable for pinpointing specific servers that may be experiencing high memory usage or other resource constraints.

In addition to the visual graphs, Ganglia also provides Hadoop metrics for each instance, categorised as distributed file system (dfs.*), Java virtual machine (jvm.*), MapReduce (mapred.*), and remote procedure calls (rpc.*). These metrics offer deeper insights into the performance of your EMR Spark application, especially when combined with the visual data.

For example, let's say you are running a memory-intensive job on your EMR Spark cluster. By enabling Ganglia, you can quickly identify whether your job is approaching the memory limits of your current instance type. If you consistently observe high memory usage, you may need to add more machines of the same instance type or upgrade to a higher-memory instance to accommodate the demands of your job.

Ganglia is an essential tool for performance optimisation and troubleshooting, providing real-time insights into the memory usage and overall health of your EMR Spark applications.

Frequently asked questions