Error evaluating HT_Memory:non_windows_generic_apache_perf

You may have seen my other blog Error evaluating omc_PerformanceOverview:non_windows_generic_apache_perf, well this is an extension to it for a different metric.

The error you see in the UI is;

Error evaluating HT_Memory:non_windows_generic_apache_perf - java.lang.Throwable: PidFile location (/etc/httpd/logs/httpd.pid) is not an absolute path for the httpd.pid file. at /home/agent/omcagent/200120.2234/plugins/oracle.em.sgfm.zip/1.47.0/scripts/generic_apache/ahs_get_memoryUsage.pl line 49.

There is an optional parameter omc_pid_file_location that defaults to omc_server_root/logs/httpd.pid.  Unfortunately, you cannot set this value using the UI.

The previous sentence is no longer valid,  the OMC UI now displays the parameter in the configure entities screen.  So you can skip to "Making sure the OMC agent user can read the PID file".

The script mentioned in the alert, which even though I am on 1.48.1 references the wrong location, which doesn’t exist has this snippet;

if ( !( -f $pidFileLoc ) ) {

    die("PidFile location ($pidFileLoc) is not an absolute path for the httpd.pid file.");

}

The perl "-f" test tests if the file is a plain file.

There are two problems with this;

1. My PID is not stored in that location.
2. My Apache is running as the user "apache" and the OMC agent doesn’t have access to the file even if it existed so it would fail for that reason.

To fix the problem we need to add the omc_pid_file_location property and make the httpd.pid file readable to the OMC agent user.

To add this property to the Apache entity update it using the omcli update_entity command.

First create a JSON payload representing your Apache entity.

$ more ~/apache.json

{

    "entities": [

        {

            "name": "MyApache",

            "type": "omc_generic_apache",

            "entityDisplayName": "MyApache",

            "properties": {

                "omc_pid_file_location": {

                    "displayName": "PID file location for httpd process",

                    "value": "/etc/httpd/run/httpd.pid"

                }

            }

        }

    ]

}

Run this command to update the entity.

$ omcli update_entity agent ~/apache.json

Oracle Management Cloud Agent

Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.

Operation Succeeded: Accepted 1 of 1 entities for processing.

Make sure that the OMC agent user can read the PID file.

$ ls -l /etc/httpd/run/httpd.pid

ls: cannot access /etc/httpd/run/httpd.pid: Permission denied

For me it turns out that the agent user cannot access /run/httpd

$ ls -ltrh /etc/httpd

total 12K

drwxr-xr-x. 2 root root 4.0K Jun 27  2019 conf.modules.d

lrwxrwxrwx. 1 root root   19 Jun 27  2019 logs -> ../../var/log/httpd

lrwxrwxrwx. 1 root root   10 Jun 27  2019 run -> /run/httpd

lrwxrwxrwx. 1 root root   29 Jun 27  2019 modules -> ../../usr/lib64/httpd/modules

drwxr-xr-x. 2 root root 4.0K Apr  3 12:50 conf

drwxr-xr-x. 2 root root 4.0K Apr 24 12:04 conf.d

$ ls -ltrh /run | grep httpd

drwx--x---.  3 root               apache              100 May  3 03:13 httpd

User setfacl to grant access.

$ sudo setfacl -m u:agentuser:x /etc/httpd/run

$ sudo setfacl -m u:agentuser:rx /etc/httpd/run/httpd.pid

You can check that everything is working fine by running;

$ omcli getMetric agent MyApache,omc_generic_apache,HT_Memory

Oracle Management Cloud Agent

Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.

memoryUsedGB,memoryUtilizationPercent,intervalSec

0.0769462585449219,0,0

Error evaluating omc_PerformanceOverview:non_windows_generic_apache_perf

Okay so fighting with Oracle Management Cloud to try and get it to monitor a generic Apache server on Oracle Linux.

Other than Apache actually being up and serving content the next thing that you're interested in is performance.

This is the error that you might see.

 Error evaluating omc_PerformanceOverview:non_windows_generic_apache_perf - java.lang.Throwable: Please check the 'server-status' configuration in the httpd.conf file at /home/omcagent/plugins/oracle.em.sgfm.zip/1.43.0/scripts/generic_apache/ahs_get_serverStatusVal.pl line 22.

The script referenced in the alert does bugger all, it has a single line of code;

die "Please check the 'server-status' configuration in the httpd.conf file";

However the metric is still collected by the agent.

omcli status agent scheduler | grep -i non_windows_generic_apache_perf
2020-04-24 11:44:01.368 : omc_generic_apache:MyApacheServer:non_windows_generic_apache_perf

And you can still generate the error on the agent.

omcli getmetric agent MyApacheServer,omc_generic_apache,omc_PerformanceOverview
Oracle Management Cloud Agent
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
EMD getmetric error: Please check the 'server-status' configuration in the httpd.conf file at /home/omcagent/200307.0800/plugins/oracle.em.sgfm.zip/1.48.0/scripts/generic_apache/ahs_get_serverStatusVal.pl line 22.

All this is despite the fact that the documentation says that performance metrics are supported.  https://docs.oracle.com/en/cloud/paas/management-cloud/monmr/apache-http-server.html

Generic Apache is such a fundamental entityType that I couldn't leave it there, after a bit of hacking in the scripts I worked out that the entity didn't have omc_version.  You can get your version from the server-status page.

Mine came back as;

<dl><dt>Server Version: Apache/2.4.6 ()</dt>

So the next problem was how to set it.  I tried editing the targets.xml file and adding

<Property NAME="omc_version" VALUE="Apache/2.4"/>

This didn't work although I expected it to.  So next I used the updateEntity REST API to add the value.

curl --location --request PATCH 'https://MyOMCInstance-MyOCITenant.omc.ocp.oraclecloud.com/serviceapi/entityModel/data/entities/AA3F54A39F7B852A73272AE687D1B638' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic c3RldmVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXTokMG55YUYwMHQkMG55YUYwMHQx' \
--header 'Content-Type: text/plain' \
--data-raw '{
    "entityType": "omc_generic_apache",
    "entityName": "Myapache",
    "properties": {
        "capability": {
            "displayName": "Capability",
            "value": "monitoring"
        },
        "omc_server_status_connect_host": {
            "displayName": "Server status connection hostname",
            "value": "localhost"
        },
        "omc_server_root": {
            "displayName": "Server Root",
            "value": "/etc/httpd"
        },
        "omc_version": {
            "displayName": "Version",
            "value": "Apache/2.4"
        },
        "omc_httpd_conf_path": {
            "displayName": "Absolute directory path of httpd configuration file",
            "value": "/etc/httpd/conf/httpd.conf"
        },
        "omc_binary_home": {
            "displayName": "Binary Home",
            "value": "/usr/sbin"
        },
        "omc_is_remote": {
            "displayName": "Is Remote",
            "value": "no"
        },
        "omc_listen_port": {
            "displayName": "Listen Port",
            "value": "80"
        },
        "host_name": {
            "displayName": "Host Name",
            "value": "MyApache.xxxxxxx.oraclevcn.com"
        },
        "omc_protocol": {
            "displayName": "Protocol",
            "value": "http"
        }
    },
    "tags": {
        "tag_a": "blah"
    },
    "entityDisplayName": "MyApache",
    "namespace":"EMAAS"
}'

you could equally have used omcli update_entity.

After adding the omc_version this is what happens.

omcli getmetric agent MyApache,omc_generic_apache,omc_PerformanceOverview
Oracle Management Cloud Agent
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
dataServedRate,idleWorkers,busyWorkers,busyWorkersUtilization
0.108,9,1,0

So now I have the metric collecting, all I need to do now is wait for them to appear in OMC.