Linux Machines: Azure Monitor Performance Preview Absence Explained

Azure Monitor is a service that collects and aggregates metrics and logs from every component of a system, providing a view of availability, performance, and resilience, and notifying users of issues. It supports the monitoring of Azure virtual machines, Azure Virtual Machine Scale Sets, hybrid machines, and physical machines running on-premises. While Azure Monitor supports Windows and Linux operating systems, there are some limitations to performance collection with VM insights for Linux machines. For instance, available memory is not available in all Linux versions, including Red Hat Linux (RHEL) 6, and metrics are only available for data disks on Linux virtual machines that use the XFS or EXT filesystem family.

Linux machine limitations in performance collection with VM insights

VM insights is a feature in Azure Monitor that allows you to monitor the performance of your virtual machines and virtual machine scale sets. It collects data on their running processes and dependencies on other resources. While VM insights supports both Windows and Linux operating systems, there are some limitations when it comes to Linux machine performance collection.

One limitation is that available memory is not available in all Linux versions, including Red Hat Linux (RHEL) 6. It will be available in Linux versions that use kernel version 3.14 or higher, and it may be available in some kernel versions between 3.0 and 3.14. Additionally, metrics are only available for data disks on Linux virtual machines that use the XFS or EXT filesystem family (EXT2, EXT3, EXT4).

Collecting performance metrics from network shared drives is also unsupported. Furthermore, if the Ethernet device for your virtual machine has more than nine characters, it won't be recognized by VM Insights, and data won't be sent to the InsightsMetrics table. The agent will, however, collect data from other sources.

It is important to note that VM insights do not support sending data to multiple Log Analytics workspaces (multi-homing).

Troubleshooting the Azure Monitor agent on Linux virtual machines

First, ensure that the Azure Monitor agent extension is installed and provisioned successfully. Open the Azure portal, select your virtual machine, then open Settings > Extensions + applications. You should see 'AzureMonitorLinuxAgent' with a status of 'Provisioning succeeded'. If not, check if your machine can reach Azure and find the extension to install using the command:

Az vm extension image list-versions --location --name AzureMonitorLinuxAgent --publisher Microsoft.Azure.Monitor

Wait for 10-15 minutes for the extension to install. If it still doesn't show up, uninstall and reinstall the extension.

Next, verify that the agent is running by checking if it is emitting heartbeat logs to the Log Analytics workspace:

Heartbeat | where Category == "Azure Monitor Agent" and Computer == "" | take 10

Skip this step if 'Custom Metrics' is the only destination in the DCR.

Check if the agent service is running with the following command:

Systemctl status azuremonitoragent

Now, verify that the Data Collection Rule (DCR) exists and is associated with the virtual machine. Open the Azure portal, select your data collection rule, then open Configuration > Resources. Your virtual machine should be listed here. If not, click 'Add' and select your VM from the resource picker. Repeat this for all DCRs.

Additionally, check if the agent was able to download the associated DCR(s) from the AMCS service by verifying the presence of the latest DCR at the following location:

/etc/opt/microsoft/azuremonitoragent/config-cache/configchunks/

If you encounter issues with Syslog collection, refer to the generic Linux AMA troubleshooting steps. Check the parsed configuration at:

/etc/opt/microsoft/azuremonitoragent/config-cache/configchunks/

Ensure that Syslog collection is defined and the log destinations match those constructed in the DCR UI/DCR JSON. If the configuration is correct, proceed to the next step. If not, the issue lies in the configuration workflow.

Examine the following files for possible configuration errors:

/var/opt/microsoft/azuremonitoragent/log/mdsd.err

/var/opt/microsoft/azuremonitoragent/log/mdsd.warn

/var/opt/microsoft/azuremonitoragent/log/mdsd.info

Finally, validate the layout of the Syslog collection workflow to ensure all necessary components are in place and accessible. For rsyslog and syslog-ng users, specific file requirements must be met. Refer to the troubleshooting documentation for detailed instructions on validating the Syslog collection workflow.

Collecting Windows and Linux performance data sources with the Log Analytics agent

Collecting performance data from Windows and Linux sources is a key function of Azure Monitor. Performance counters in Windows and Linux provide insights into the performance of hardware components, operating systems, and applications. Azure Monitor can collect performance counter data from Log Analytics agents at frequent intervals for near real-time analysis, and can also aggregate performance data for longer-term analysis and reporting.

To collect Windows and Linux performance data sources with the Log Analytics agent, you will need to configure performance counters. This can be done from the Legacy agents management menu for the Log Analytics workspace. When configuring Windows or Linux performance counters for a new workspace, you can quickly create several common counters. Ensure any counters you want to initially create are selected, then select "Add the selected performance counters".

For Windows performance counters, you can choose a specific instance for each performance counter. For Linux performance counters, the instance of each counter that you choose will apply to all child counters of the parent counter.

• Select "Add performance counter".
• Enter the name of the counter in the text box in the format object(instance)\counter. When you start typing, a matching list of common counters will appear. You can either select a counter from the list or enter a custom one. You can also return all instances for a particular counter by specifying object\counter.
• When adding a counter, the default Sample Interval is 10 seconds. You can change this to a higher value of up to 1,800 seconds (30 minutes) if you want to reduce the storage requirements of the collected performance data.
• After you're finished adding counters, select "Apply" at the top of the screen to save the configuration.

For Linux performance counters, follow these steps:

• Select "Add performance counter".
• Enter the name of the counter in the text box in the format object(instance)\counter. As with Windows, a matching list of common counters will appear as you type. You can select from this list or enter your own.
• All counters for a Linux object use the same Sample Interval, which defaults to 10 seconds. As with Windows, you can change this default value to a higher value of up to 1,800 seconds (30 minutes) to reduce the storage requirements of the collected performance data.
• After you're finished adding counters, select "Apply" to save the configuration.

Instead of using the Azure portal to configure Linux performance counters, you can edit configuration files on the Linux agent. Performance metrics to collect are controlled by the configuration in /etc/opt/microsoft/omsagent//conf/omsagent.conf. Each object, or category, of performance metrics to collect should be defined in the configuration file as a single element. The syntax follows the pattern:

<source> type oms_omi object_name "Processor" instance_regex ".*" counter_name_regex ".*" interval 30s </source>

The parameters in this element include the object name for the collection, a regular expression that defines which instances to collect, a regular expression that defines which counters to collect, and the frequency at which the object's counters are collected.

Azure Monitor collects all specified performance counters at their specified sample intervals on all agents that have that counter installed. The data is not aggregated, and the raw data is available in all log query views for the duration specified by your Log Analytics workspace.

Deploying the Azure Monitor agent to Azure and hybrid virtual machines

The Azure Monitor agent is a tool that collects monitoring data from the guest operating system of Azure and hybrid virtual machines and delivers it to Azure Monitor for use by features, insights, and other services such as Microsoft Sentinel and Microsoft Defender for Cloud. The agent is installed on virtual machines running in Azure, in other clouds, or on-premises, where it has access to local logs and performance data.

The Azure Monitor agent is implemented as a virtual machine extension, so you can install it using a variety of standard methods, including PowerShell, CLI, and Resource Manager templates. It is also possible to install the agent on a single machine or at scale using Azure Policy or other tools. In some cases, the agent will be automatically installed when you enable a feature that requires it, such as Microsoft Sentinel.

• Create a Log Analytics workspace: While a Log Analytics workspace is not required to deploy the Azure Monitor agent, it is necessary to collect the data that it sends. There is no cost for the workspace, but there are ingestion and retention costs when you collect data. It is common for many environments to use a single workspace for all their virtual machines and other Azure resources they monitor.
• Prepare hybrid machines: A hybrid machine is any machine not running in Azure. It could be a virtual machine running in another cloud or hosted provider, or a virtual or physical machine running on-premises in your data center. Use Azure Arc-enabled servers on hybrid machines so you can manage them similarly to your Azure virtual machines.
• Understand network requirements: The Azure Monitor agent for both Linux and Windows communicates outbound to the Azure Monitor service over TCP port 443. The Dependency agent uses the Azure Monitor agent for all communication and doesn't require any additional ports.
• Choose agent deployment options: As mentioned earlier, the Azure Monitor agent can be installed using a variety of methods, including PowerShell, CLI, and Resource Manager templates.
• Production deployment at scale: If you have a significant number of virtual machines, you should deploy the agent using Azure Policy. This will ensure that the agent is automatically added to existing virtual machines and any new ones that you deploy.
• Testing and simple deployments: When you create a data collection rule in the Azure portal, you can specify virtual machines to receive it. The Azure Monitor agent will be automatically installed on any machines that don't already have it.
• Testing and simple deployments with pre-configured monitoring: VM insights provide simplified onboarding of agents in the Azure portal. With a single click for a particular machine, it installs the Azure Monitor agent, connects to a workspace, and starts collecting performance data.
• Windows client installer: Use the Windows client installer to install the agent on Windows clients such as Windows 11.

The Azure Monitor agent replaces legacy agents, such as the Log Analytics agent and the Azure Diagnostic extension, which are still available but should only be used if you require particular functionality not yet available with the Azure Monitor agent.

Configuring data collection rules for the Azure Monitor agent

Data Collection Rules (DCRs) are a part of an ETL-like data collection process that improves on legacy data collection methods for Azure Monitor. This process uses a common data ingestion pipeline, the Azure Monitor pipeline, for all data sources and a standard method of configuration that is more manageable and scalable than other methods.

DCRs are used to specify the data to be collected from Azure resources and the configuration for how that data is collected. They are stored in Azure so they can be centrally deployed and managed like any other Azure resource. They are sets of instructions supporting data collection using the Azure Monitor pipeline.

When creating a DCR in the Azure portal, users are walked through a series of pages to provide the information needed to collect data from the machines they specify. This includes:

• The machines that will use the DCR. When a machine is added to the DCR, it creates a data collection rule association (DCRA) between the machine and the DCR.
• The type of data to collect from the machine.
• The destination where the collected data should be sent.

Once a DCR has been created and associated with a machine, it is possible to verify that the agent is operational and that data is being collected by running queries in the Log Analytics workspace.

To send data across tenants, you must first enable Azure Lighthouse.

Frequently asked questions