Host Monitoring Nozzle: Missing From Scm, Now What?

The error message Could not find a HOST_MONITORING nozzle from SCM is related to issues with the Cloudera Manager (CM) and the installation of CDH5. This problem results in monitoring issues, with all services displaying Unknown Health and charts showing the error Unable to issue query: could not connect to the Host Monitor. The solution to this issue may be as simple as restarting the Cloudera management services or creating the directories /var/lib/cloudera-service-monitor and /var/lib/cloudera-host-monitor manually.

Restart Cloudera management services

To restart the Cloudera Management Service, you must have the Limited Cluster Administrator role or higher. Then, follow these steps:

• Select Clusters > Cloudera Management Service.
• Select Actions > Restart.
• On the Home > Status tab, click the options menu to the right of Cloudera Management Service and select Restart.
• Click Restart again to confirm. You will see the progress of the restart in the Command Details window.
• When the Command completed with n/n successful subcommands message appears, the task is complete. Click Close.

Alternatively, you can restart Cloudera Manager Agents by using the following commands:

To stop or restart agents while leaving managed processes running:

Sudo service cloudera-scm-agent stop

Sudo service cloudera-scm-agent restart

To hard stop or restart agents (this will kill all running managed service processes on the host(s) where the command is run):

Sudo service cloudera-scm-agent hard_stop

Sudo service cloudera-scm-agent hard_restart

Note that hard stopping and restarting agents requires you to stop all roles on the host. If you cannot stop all roles immediately, you must do so within 60 days of the hard stop or restart.

If you are encountering issues with your Cloudera services, such as the "Could not find a HOST_MONITORING nozzle from SCM" error, restarting your Cloudera management services may resolve the problem.

Check the Health Service

To check the health service, you can refer to the logs and warnings. Here are some of the warnings you may encounter:

• WARN [1752771961@scm-web-7:cmf.StatusProvider@503] Failed to get status.
• WARN [1202997703@scm-web-1:tsquery.TimeSeriesQueryService@503] Could not find a HOST_MONITORING nozzle from SCM.
• WARN [44428231@scm-web-34:cmf.EventsController@503] Exception querying events com.cloudera.cmon.MgmtServiceLocatorException: There is no configured event store.

If you encounter these warnings, you can try restarting your Cloudera management services. This may resolve the issue and get the monitoring working again.

If you are still facing issues, you can try checking the following:

• Ensure that you have installed and configured the necessary components, such as cloudera-scm-server and cloudera-scm-agent.
• Verify the Cloudera management services are running and functional.
• Review the service and host monitor logs for any additional errors or issues.
• Check if there are any other applications using the same App Service plan as the instance that may be impacting its health.

Confirm antivirus exclusions are configured correctly

To confirm that antivirus exclusions are configured correctly, you can follow these steps:

• Identify the specific antivirus (AV) product installed on your system. This could be Windows Defender, McAfee EPO, or another AV solution.
• Check the documentation for your AV product to determine the recommended file and folder exclusions. These exclusions ensure that the AV product does not interfere with the normal operation of other software, such as the Carbon Black EDR sensor.
• Create custom rules or exclusions in your AV product to exclude the recommended files and folders from scanning. This may involve adding specific paths or using a trailing asterisk () to signify all directory contents.
• Verify that the exclusions are applied correctly by checking the AV product's settings or configuration files.
• Monitor the system's performance and behavior to ensure that there are no unexpected issues or errors related to the exclusions.
• /var/opt/carbonblack/
• /Applications/Confer.app/
• /Applications/VMware Carbon Black Cloud
• /Library/Application Support/com.vmware.carbonblack.cloud/
• /Library/Extensions/CbDefenseSensor.kext
• C:\Program Files\Confer\
• C:\ProgramData\CarbonBlack\
• C:\Windows\System32\drivers\ctifile.sys
• C:\Windows\system32\CbAMSI.dll
• /etc/init.d/cbdaemon
• /opt/carbonblack/response/

By confirming that antivirus exclusions are configured correctly, you can avoid potential interoperability problems and ensure that your system functions as intended.

Check for network issues

Checking for network issues is a crucial step in troubleshooting and can help identify problems with your connection. Here are some detailed steps to help you check for network issues:

Check the Hardware:

• Ensure all your hardware components, such as routers, modems, switches, and cables, are properly connected and functioning.
• Power cycle your modem, router, and PC by turning them off and on again. Leave each device off for at at least 60 seconds before powering them back on.

Use Ping and Tracert:

• Use the "ping" command to send a request to a well-known server, such as Google's DNS servers ("ping 8.8.8.8"). This will help determine if your router is working correctly.
• Utilize the "tracert" command ("tracert 8.8.8.8") to identify each step or "hop" between your router and the destination server. This will help you locate where the error is occurring.

Perform a DNS Check:

• Use the "nslookup" command to check if there are issues with the server you're trying to connect to.
• If you receive results such as "Timed Out," "Server Failure," or "No Response from Server," it may indicate a problem with the DNS server.

Check Virus and Malware Protection:

Ensure your virus and malware protection tools are running correctly and haven't flagged any issues that could be affecting your network.

Review Database Logs:

Analyze your database logs to ensure the databases are functioning as expected. Database issues can impact network performance, so it's important to review these logs for any signs of trouble.

By following these steps, you can effectively check for network issues and narrow down the potential causes of your connection problems.

Manually create appropriate Service Principal Names (SPNs)

Manually creating appropriate Service Principal Names (SPNs) involves several steps and considerations. Here is a detailed guide:

Understanding SPNs

SPNs are unique identifiers for service instances, used by Kerberos authentication to associate a service instance with a service login account. SPNs are recorded in an Active Directory (AD) database, showing which services are registered to specific accounts. The core of Kerberos is the Key Distribution Center (KDC) services, which use 88/TCP and 88/UDP ports.

Tools for Managing SPNs

The setspn command-line tool, built into Windows Server 2008, is used to manage SPNs. To use setspn, you must have the Active Directory Domain Services (AD DS) server role installed and run the command from an elevated command prompt. Setspn can be used to add, reset, and delete SPNs.

Permissions Required to Modify SPNs

Modifying SPNs requires specific permissions, typically granted to domain administrators but delegable to other accounts. The required permissions are:

• Write ServicePrincipalName permission: Allows modification of SPNs and can be assigned using the Security tab in the Active Directory Users and Computers snap-in or the dsacls command-line tool.
• Validated Write to Service Principal Name permission: Enables a service to register its own SPN dynamically and is typically used with service accounts.
• Self permission: Allows a service account to modify its own SPNs, provided it also has the Write ServicePrincipalName and Validated Write permissions.

Registering an SPN Manually

To register an SPN manually, use the Microsoft-provided Setspn.exe utility. You need domain admin privileges or appropriate rights to run this tool and register an SPN. The -s option prevents creating existing SPN definitions.

Syntax for Adding, Deleting, and Resetting SPNs

To add an SPN, use the command: setspn -s service/name hostname, where service/name is the desired SPN and hostname is the computer object's actual hostname.

To delete an SPN, use the command: setspn -d service/name hostname, where service/name is the SPN to remove, and hostname is the actual hostname.

To reset the default SPN values, use the command: setspn -r hostname, where hostname is the actual hostname.

Viewing Current SPNs

To view the current SPNs associated with a specific account, use the command: setspn -L accountname or setspn -L localhost.

Handling Duplicate SPNs

Duplicate SPNs can cause authentication issues. To check for duplicates, use the -X switch with setspn, which will scan the entire domain and report any duplicates.

Frequently asked questions