Wednesday, 6 May 2020

Error evaluating HT_Memory:non_windows_generic_apache_perf


You may have seen my other blog Error evaluating omc_PerformanceOverview:non_windows_generic_apache_perf, well this is an extension to it for a different metric.

The error you see in the UI is;

Error evaluating HT_Memory:non_windows_generic_apache_perf - java.lang.Throwable: PidFile location (/etc/httpd/logs/httpd.pid) is not an absolute path for the httpd.pid file. at /home/agent/omcagent/200120.2234/plugins/oracle.em.sgfm.zip/1.47.0/scripts/generic_apache/ahs_get_memoryUsage.pl line 49.

There is an optional parameter omc_pid_file_location that defaults to omc_server_root/logs/httpd.pid.  Unfortunately, you cannot set this value using the UI.

The previous sentence is no longer valid,  the OMC UI now displays the parameter in the configure entities screen.  So you can skip to "Making sure the OMC agent user can read the PID file".


The script mentioned in the alert, which even though I am on 1.48.1 references the wrong location, which doesn’t exist has this snippet;

if ( !( -f $pidFileLoc ) ) {
    die("PidFile location ($pidFileLoc) is not an absolute path for the httpd.pid file.");
}

The perl "-f" test tests if the file is a plain file.

There are two problems with this;
  1. My PID is not stored in that location.
  2. My Apache is running as the user "apache" and the OMC agent doesn’t have access to the file even if it existed so it would fail for that reason.

To fix the problem we need to add the omc_pid_file_location property and make the httpd.pid file readable to the OMC agent user.

To add this property to the Apache entity update it using the omcli update_entity command.

First create a JSON payload representing your Apache entity.

$ more ~/apache.json
{
    "entities": [
        {
            "name": "MyApache",
            "type": "omc_generic_apache",
            "entityDisplayName": "MyApache",
            "properties": {
                "omc_pid_file_location": {
                    "displayName": "PID file location for httpd process",
                    "value": "/etc/httpd/run/httpd.pid"
                }
            }
        }
    ]
}

Run this command to update the entity.

$ omcli update_entity agent ~/apache.json
Oracle Management Cloud Agent
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
Operation Succeeded: Accepted 1 of 1 entities for processing.

Make sure that the OMC agent user can read the PID file.

$ ls -l /etc/httpd/run/httpd.pid
ls: cannot access /etc/httpd/run/httpd.pid: Permission denied

For me it turns out that the agent user cannot access /run/httpd

$ ls -ltrh /etc/httpd
total 12K
drwxr-xr-x. 2 root root 4.0K Jun 27  2019 conf.modules.d
lrwxrwxrwx. 1 root root   19 Jun 27  2019 logs -> ../../var/log/httpd
lrwxrwxrwx. 1 root root   10 Jun 27  2019 run -> /run/httpd
lrwxrwxrwx. 1 root root   29 Jun 27  2019 modules -> ../../usr/lib64/httpd/modules
drwxr-xr-x. 2 root root 4.0K Apr  3 12:50 conf
drwxr-xr-x. 2 root root 4.0K Apr 24 12:04 conf.d

$ ls -ltrh /run | grep httpd
drwx--x---.  3 root               apache              100 May  3 03:13 httpd

User setfacl to grant access.

$ sudo setfacl -m u:agentuser:x /etc/httpd/run
$ sudo setfacl -m u:agentuser:rx /etc/httpd/run/httpd.pid

You can check that everything is working fine by running;

$ omcli getMetric agent MyApache,omc_generic_apache,HT_Memory
Oracle Management Cloud Agent
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
memoryUsedGB,memoryUtilizationPercent,intervalSec
0.0769462585449219,0,0