Monitoring Ec2 Memory Usage: A Comprehensive Guide

Monitoring memory usage on an EC2 instance is not a straightforward process, especially for those new to AWS. While metrics like CPU load logs and network latency are available by default on the AWS monitoring dashboard, EC2 memory usage metrics are not. This means that to monitor EC2 memory usage, you will need to install and configure a CloudWatch agent on your instance.

The first step is to create an EC2 role with permission to access CloudWatch. This can be done through the Identity and Access Management (IAM) dashboard. Once the role is created, it can be attached to any EC2 instance that requires monitoring.

Next, you will need to SSH into the EC2 instance and download the appropriate CloudWatch agent for your instance's operating system. For example, for an Amazon Linux OS, the command is:

> wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm

Once downloaded, the agent can be installed, and then configured by creating a file at /opt/aws/amazon-cloudwatch-agent/bin/config.json. This file specifies the metric to be monitored (in this case, memory usage) and the monitoring interval.

Finally, the CloudWatch agent can be started, and the memory usage metrics can be visualised on the CloudWatch dashboard in the AWS console.

Installing CloudWatch Agent

To install the CloudWatch agent, you must first establish internet connectivity in your EC2 instance. Then, you can create an IAM role to run the agent on your EC2 instance.

• Open the AWS Identity and Access Management (IAM) console.
• In the navigation pane, choose 'Roles'.
• Choose 'Create role'.
• For 'Choose the service that will use this role', select 'EC2'.
• Click 'Next: Permissions'.
• Select the 'CloudWatchAgentServerPolicy' check box in the list of policies.
• Click 'Next: Tags', and then 'Next: Review'.
• Enter a name for the role, such as 'CloudWatchAgentServerRole'.
• (Optional) Provide a role description.
• Confirm that 'CloudWatchAgentServerPolicy' appears next to 'Policies'.
• Click 'Create role'.
• Attach the new IAM role to the EC2 instance.

Once you have created the IAM role, you can download and install the CloudWatch agent.

Amazon Linux 2023 and Amazon Linux 2:

To download the CloudWatch agent package, enter the following command:

> wget https://s3.region.amazonaws.com/amazoncloudwatch-agent-region/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm

Replace `region` with your AWS Region.

To install the CloudWatch agent, run the following command:

> sudo rpm -U ./amazon-cloudwatch-agent.rpm

Windows Server:

To download the CloudWatch agent package, enter the following command:

> https://s3.region.amazonaws.com/amazoncloudwatch-agent-region/windows/amd64/latest/amazon-cloudwatch-agent.msi

Replace `region` with your AWS Region.

Move to the directory containing the package, and then enter the following command:

> msiexec /i amazon-cloudwatch-agent.msi

Ubuntu:

To download the CloudWatch agent package, enter the following command:

> wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

To install the CloudWatch agent, run the following command:

> sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

After installing the CloudWatch agent, you can start the agent using the command line. Here is an example command to start the CloudWatch agent on an EC2 instance running Linux:

> sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path

Replace `configuration-file-path` with the path to your agent configuration file.

Creating IAM roles

To monitor EC2 memory usage, you need to install and configure a CloudWatch agent on each EC2 instance. This process involves several steps, including creating an IAM role with the necessary permissions. Here's a detailed guide on creating IAM roles for EC2 instances to monitor memory usage:

• Go to the Identity and Access Management (IAM) dashboard in your AWS console.
• Click on 'Roles' in the side menu and then select the 'Create Role' button.
• On the next page, select 'EC2' as the service that will require permissions and click 'Next'.
• In the search box, type "CloudWatchFullAccess" and select the checkbox next to it. This gives the EC2 instance access to CloudWatch. Click 'Next'.
• (Optional) You can add tags for better organization if needed.
• Provide a descriptive name and a short description for the role.
• Click 'Create Role'.

Now that the role is created, you can attach it to any EC2 instance you want to monitor for memory usage.

Attaching the Role to an EC2 Instance:

• Go to the EC2 dashboard in your AWS console.
• Click on the instance you wish to attach the newly created role to.
• Click on 'Actions' > 'Security' > 'Modify IAM Role'.
• Select the newly created role from the drop-down menu and click 'Apply'.

With these steps completed, your EC2 instance now has the necessary IAM role attached, and you can proceed to install and configure the CloudWatch agent.

Installing and Configuring the CloudWatch Agent:

Detailed steps for this section can be found in the sources provided.)

• SSH into the EC2 instance.
• Download the appropriate CloudWatch agent for your instance's operating system.
• Install the CloudWatch agent.
• Create a configuration file that specifies the metric to be monitored (memory usage) and the monitoring interval.
• Start the CloudWatch agent with the provided command.

Once the agent is running, you can visualize your EC2 memory usage data in the CloudWatch dashboard.

By following these steps, you will be able to effectively monitor the memory usage of your EC2 instances, allowing you to improve performance, detect issues, and make informed decisions about your AWS infrastructure.

Downloading the agent package

To download the agent package, you will need to find the appropriate download link for your instance's operating system. For example, if you are using an Amazon Linux OS, the command to download the agent package is as follows:

`wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm`

For other operating systems, the download link can be found here. Once you have identified the correct download link for your operating system, simply replace the link in the following command:

`wget download-link`

After running this command, the agent package will be downloaded to your instance. You can then proceed to install the agent package.

It is important to note that the CloudWatch agent is supported on both Windows and Linux systems. Additionally, you have the option to download and install the agent manually using the AWS CLI, or you can integrate it with the AWS Systems Manager Agent (SSM Agent).

Creating the CloudWatch agent configuration file

To create the CloudWatch agent configuration file, you need to specify the metrics you want to collect and the frequency of data collection.

First, create a file at /opt/aws/amazon-cloudwatch-agent/bin/config.json using your preferred method and paste in the following configuration:

{ "metrics":{ "metrics_collected":{ "mem":{ "measurement": [ "mem_used_percent" ], "metrics_collection_interval":30 } } } }

This configuration file tells the agent to collect memory usage data in percentage format every 30 seconds. You can, of course, specify whatever interval you prefer.

If you are using a Windows server, use this path instead: $Env:ProgramDataAmazonAmazonCloudWatchAgentamazon-cloudwatch-agent.json.

{ "metrics": { "metrics_collected": { "mem": { "measurement": [ "mem_used_percent" ] }, "disk": { "measurement": [ "used_percent" ], "resources": [ "*" ] } }, "append_dimensions": { "InstanceId": "${aws:InstanceId}" } } }

This configuration file includes both memory and disk usage metrics, with the memory usage metric specified as "mem_used_percent" and the disk usage metric specified as "used_percent". The "resources": ["*"] field indicates that the agent should collect data from all available resources.

For a Windows system, the configuration file would look like this:

{ "metrics": { "metrics_collected": { "LogicalDisk": { "measurement": [ "% Free Space" ], "resources": [ "*" ] }, "Memory": { "measurement": [ "% Committed Bytes In Use" ] } }, "append_dimensions": { "InstanceId": "${aws:InstanceId}" } } }

In this case, the agent is collecting the percentage of free space on the logical disk and the percentage of committed bytes in use in memory.

Once you have created and configured the CloudWatch agent configuration file, you can start the agent with the following command:

Sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

Starting the agent

With the agent installed, you can now start it. Use the following command to start the CloudWatch agent:

Bash

Sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s

Or, for Windows:

Bash

\amazon-cloudwatch-agent-ctl.bat -a fetch-config -m ec2 -c file:C:\ProgramData\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.json

If you are using an older version of the CloudWatch agent, you may need to use the following command:

Bash

Sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s

Once the agent is started, it will begin collecting metrics and sending them to CloudWatch. You can verify that the agent is running by checking the status:

Bash

Sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

This will display the current status of the CloudWatch agent, including whether it is running or stopped.

It is important to note that the CloudWatch agent runs as a daemon process, which means it will continue to run in the background even after you log out of the instance. The agent will also automatically start upon system boot, so you don't need to manually start it each time the instance restarts.

You can also customize the metrics that the agent collects and the frequency of data collection. To do so, you can modify the configuration file that you created earlier. For example, you can change the "mem_used_percent" metric to "mem_used_bytes" to collect the memory usage in bytes instead of percentage.

Additionally, you can specify multiple metrics to be collected by adding them to the "measurement" array in the configuration file. For example:

Json

"measurement": ["mem_used_percent", "mem_used_bytes"]

This will collect both the memory usage percentage and the memory usage in bytes.

You can also adjust the "metrics_collection_interval" value to change how often the agent collects and reports metrics. For example, setting it to 60 will collect and report metrics every 60 seconds.

Keep in mind that collecting and reporting metrics too frequently may impact the performance of your instance, so it is important to find a balance that suits your monitoring needs without affecting the performance of your application.

Once you have made any desired changes to the configuration file, you will need to restart the agent for the changes to take effect:

Bash

Sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop

Sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start

This will stop and then start the agent, applying the new configuration settings.

It is worth mentioning that you can also use other tools and services to monitor your EC2 instances, such as AWS Systems Manager, which provides a unified interface for managing and monitoring your AWS resources. Additionally, there are third-party tools and services available that offer more advanced monitoring and visualisation capabilities.

However, in this guide, we focused on using the CloudWatch agent as it is a simple and effective way to collect and monitor memory usage on your EC2 instances, providing valuable insights into the performance and health of your applications.

Frequently asked questions