Reset OMC Log Analytics file collections

Developing a series of log analytics log parsers and log sources is trickey at the best of times.  Rather than using the on-demand upload I wanted the Oracle Management Cloud agent to pick up the files.

As is almost always the case I had to delete the data and try again, several times.  If you don't delete the data first you will end up with duplicate log records.

Deleting the data was easy.

First of all remove the association to the entity that the log source is tied to, then goto the Log Admin section and define a query to delete all the previously uploaded data, in my case that was:-

'Log Source'='OIC Connectivity Agent Log Source'

Unfortunately log analytics has a bunch of internal state files that keep a record of what has already been processed.

These files are contained in $AGENT_INST/sysman/emd/state/loganalytics/os_file and contain data similar to:-

version=154

dirName=/home/oracle/OICS_CONNECTIVITYAGENT02/agenthome/logs

isLargeDir=false

forwardTime=0

forwardFileName=null

backwardTime=0

backwardFileName=null

lastReadCycleTime=1631087347454

################################

agent-diagnostic7.log;80467459;1629238676000;10485951;10485951;ISO-8859-1;UTC;2021-08-14T14:12:42.991Z;0

agent-diagnostic5.log;80467469;1629814398000;10485780;10485780;ISO-8859-1;UTC;2021-08-21T06:15:54.435Z;0

agent-diagnostic6.log;80467468;1629526554000;10486076;10486076;ISO-8859-1;UTC;2021-08-17T22:17:56.123Z;0

agent-diagnostic8.log;80467467;1628950362000;10485898;10485898;ISO-8859-1;UTC;2021-08-11T05:50:17.475Z;0

agent-diagnostic3.log;80467462;1630389762000;10485783;10485783;ISO-8859-1;UTC;2021-08-27T22:07:46.938Z;0

agent-diagnostic4.log;80467470;1630102066000;10485844;10485844;ISO-8859-1;UTC;2021-08-24T14:13:18.831Z;0

agent-diagnostic9.log;80467466;1628661017000;10485860;10485860;ISO-8859-1;UTC;2021-08-09T04:44:02.439Z;0

agent-diagnostic0.log;80467465;1631087314000;5765858;5765858;ISO-8859-1;UTC;2021-09-06T21:52:22.778Z;0

agent-diagnostic1.log;80467464;1630965142000;10485953;10485953;ISO-8859-1;UTC;2021-09-03T13:57:31.951Z;0

agent-diagnostic2.log;80467463;1630677451000;10485791;10485791;ISO-8859-1;UTC;2021-08-31T06:02:42.259Z;0

agent-upgrade.log;69530634;1631011778000;552233;552233;ISO-8859-1;UTC;2021-07-31T19:49:30.218Z;0

agent-audit.log.1;69832466;1625572011000;6500;6500;ISO-8859-1;UTC;2021-01-06T14:37:06.813Z;0

agent-alert.log;69530630;1631033124000;734154;734154;ISO-8859-1;UTC;2021-08-01T17:24:50.482Z;0

agent-audit.log;69530632;1626101136000;553854;553854;ISO-8859-1;UTC;2020-08-21T11:15:12.950Z;0

Stop the OMC Agent using omcli stop agent

Delete the file associated for your log source

Start the OMC Agent using omcli start agent

Re-Associate the entity using the OMC console.

Error evaluating HT_Memory:non_windows_generic_apache_perf

You may have seen my other blog Error evaluating omc_PerformanceOverview:non_windows_generic_apache_perf, well this is an extension to it for a different metric.

The error you see in the UI is;

Error evaluating HT_Memory:non_windows_generic_apache_perf - java.lang.Throwable: PidFile location (/etc/httpd/logs/httpd.pid) is not an absolute path for the httpd.pid file. at /home/agent/omcagent/200120.2234/plugins/oracle.em.sgfm.zip/1.47.0/scripts/generic_apache/ahs_get_memoryUsage.pl line 49.

There is an optional parameter omc_pid_file_location that defaults to omc_server_root/logs/httpd.pid.  Unfortunately, you cannot set this value using the UI.

The previous sentence is no longer valid,  the OMC UI now displays the parameter in the configure entities screen.  So you can skip to "Making sure the OMC agent user can read the PID file".

The script mentioned in the alert, which even though I am on 1.48.1 references the wrong location, which doesn’t exist has this snippet;

if ( !( -f $pidFileLoc ) ) {

    die("PidFile location ($pidFileLoc) is not an absolute path for the httpd.pid file.");

}

The perl "-f" test tests if the file is a plain file.

There are two problems with this;

1. My PID is not stored in that location.
2. My Apache is running as the user "apache" and the OMC agent doesn’t have access to the file even if it existed so it would fail for that reason.

To fix the problem we need to add the omc_pid_file_location property and make the httpd.pid file readable to the OMC agent user.

To add this property to the Apache entity update it using the omcli update_entity command.

First create a JSON payload representing your Apache entity.

$ more ~/apache.json

{

    "entities": [

        {

            "name": "MyApache",

            "type": "omc_generic_apache",

            "entityDisplayName": "MyApache",

            "properties": {

                "omc_pid_file_location": {

                    "displayName": "PID file location for httpd process",

                    "value": "/etc/httpd/run/httpd.pid"

                }

            }

        }

    ]

}

Run this command to update the entity.

$ omcli update_entity agent ~/apache.json

Oracle Management Cloud Agent

Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.

Operation Succeeded: Accepted 1 of 1 entities for processing.

Make sure that the OMC agent user can read the PID file.

$ ls -l /etc/httpd/run/httpd.pid

ls: cannot access /etc/httpd/run/httpd.pid: Permission denied

For me it turns out that the agent user cannot access /run/httpd

$ ls -ltrh /etc/httpd

total 12K

drwxr-xr-x. 2 root root 4.0K Jun 27  2019 conf.modules.d

lrwxrwxrwx. 1 root root   19 Jun 27  2019 logs -> ../../var/log/httpd

lrwxrwxrwx. 1 root root   10 Jun 27  2019 run -> /run/httpd

lrwxrwxrwx. 1 root root   29 Jun 27  2019 modules -> ../../usr/lib64/httpd/modules

drwxr-xr-x. 2 root root 4.0K Apr  3 12:50 conf

drwxr-xr-x. 2 root root 4.0K Apr 24 12:04 conf.d

$ ls -ltrh /run | grep httpd

drwx--x---.  3 root               apache              100 May  3 03:13 httpd

User setfacl to grant access.

$ sudo setfacl -m u:agentuser:x /etc/httpd/run

$ sudo setfacl -m u:agentuser:rx /etc/httpd/run/httpd.pid

You can check that everything is working fine by running;

$ omcli getMetric agent MyApache,omc_generic_apache,HT_Memory

Oracle Management Cloud Agent

Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.

memoryUsedGB,memoryUtilizationPercent,intervalSec

0.0769462585449219,0,0

Error evaluating omc_PerformanceOverview:non_windows_generic_apache_perf

Okay so fighting with Oracle Management Cloud to try and get it to monitor a generic Apache server on Oracle Linux.

Other than Apache actually being up and serving content the next thing that you're interested in is performance.

This is the error that you might see.

 Error evaluating omc_PerformanceOverview:non_windows_generic_apache_perf - java.lang.Throwable: Please check the 'server-status' configuration in the httpd.conf file at /home/omcagent/plugins/oracle.em.sgfm.zip/1.43.0/scripts/generic_apache/ahs_get_serverStatusVal.pl line 22.

The script referenced in the alert does bugger all, it has a single line of code;

die "Please check the 'server-status' configuration in the httpd.conf file";

However the metric is still collected by the agent.

omcli status agent scheduler | grep -i non_windows_generic_apache_perf
2020-04-24 11:44:01.368 : omc_generic_apache:MyApacheServer:non_windows_generic_apache_perf

And you can still generate the error on the agent.

omcli getmetric agent MyApacheServer,omc_generic_apache,omc_PerformanceOverview
Oracle Management Cloud Agent
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
EMD getmetric error: Please check the 'server-status' configuration in the httpd.conf file at /home/omcagent/200307.0800/plugins/oracle.em.sgfm.zip/1.48.0/scripts/generic_apache/ahs_get_serverStatusVal.pl line 22.

All this is despite the fact that the documentation says that performance metrics are supported.  https://docs.oracle.com/en/cloud/paas/management-cloud/monmr/apache-http-server.html

Generic Apache is such a fundamental entityType that I couldn't leave it there, after a bit of hacking in the scripts I worked out that the entity didn't have omc_version.  You can get your version from the server-status page.

Mine came back as;

<dl><dt>Server Version: Apache/2.4.6 ()</dt>

So the next problem was how to set it.  I tried editing the targets.xml file and adding

<Property NAME="omc_version" VALUE="Apache/2.4"/>

This didn't work although I expected it to.  So next I used the updateEntity REST API to add the value.

curl --location --request PATCH 'https://MyOMCInstance-MyOCITenant.omc.ocp.oraclecloud.com/serviceapi/entityModel/data/entities/AA3F54A39F7B852A73272AE687D1B638' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic c3RldmVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXTokMG55YUYwMHQkMG55YUYwMHQx' \
--header 'Content-Type: text/plain' \
--data-raw '{
    "entityType": "omc_generic_apache",
    "entityName": "Myapache",
    "properties": {
        "capability": {
            "displayName": "Capability",
            "value": "monitoring"
        },
        "omc_server_status_connect_host": {
            "displayName": "Server status connection hostname",
            "value": "localhost"
        },
        "omc_server_root": {
            "displayName": "Server Root",
            "value": "/etc/httpd"
        },
        "omc_version": {
            "displayName": "Version",
            "value": "Apache/2.4"
        },
        "omc_httpd_conf_path": {
            "displayName": "Absolute directory path of httpd configuration file",
            "value": "/etc/httpd/conf/httpd.conf"
        },
        "omc_binary_home": {
            "displayName": "Binary Home",
            "value": "/usr/sbin"
        },
        "omc_is_remote": {
            "displayName": "Is Remote",
            "value": "no"
        },
        "omc_listen_port": {
            "displayName": "Listen Port",
            "value": "80"
        },
        "host_name": {
            "displayName": "Host Name",
            "value": "MyApache.xxxxxxx.oraclevcn.com"
        },
        "omc_protocol": {
            "displayName": "Protocol",
            "value": "http"
        }
    },
    "tags": {
        "tag_a": "blah"
    },
    "entityDisplayName": "MyApache",
    "namespace":"EMAAS"
}'

you could equally have used omcli update_entity.

After adding the omc_version this is what happens.

omcli getmetric agent MyApache,omc_generic_apache,omc_PerformanceOverview
Oracle Management Cloud Agent
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
dataServedRate,idleWorkers,busyWorkers,busyWorkersUtilization
0.108,9,1,0

So now I have the metric collecting, all I need to do now is wait for them to appear in OMC.

Oracle Integration Cloud Log Analytics Audit Log Parser

At the time of writing Oracle Management Cloud currently comes with a Log Parser for Oracle Integration Cloud Audit logs (ics-audit.log).

The parser does not work if the userId contains a word separating characters like an email address.

The current parser definition is

\[{TIMEDATE}\]\s+\[(\w+)\]\s+\[(\w+)\]\s*\[(.*)?\]\s+\[(\S+)\]\s+\[\w+\:\s+(\d+)\]\s*\[\w+\:\s*(?:\<)?(\w+)(?:\>)?\]\s+\[\w+\:\s+(\S+)\]\s*\[\w+\:\s*([^\,]*)?\]\s*\[\w+\-\w+\:\s*(\w*)?\]\s*\[\w+\-\w+\:\s*(\w*)?\]\s+\[([^\,]*)\]\:\s*\[([^\,]*)?\,([^\,]*)?\,([^\,]*)?\,([^\,]*)?\,([^\,]*)?\,([^\,]*)?\](.*)?

The modified parser definition is

\[{TIMEDATE}\]\s+\[(\w+)\]\s+\[(\w+)\]\s*\[(.*)?\]\s+\[(\S+)\]\s+\[\w+\:\s+(\d+)\]\s*\[\w+\:\s*(?:\<)?(.*)(?:\>)?\]\s+\[\w+\:\s+(\S+)\]\s*\[\w+\:\s*([^\,]*)?\]\s*\[\w+\-\w+\:\s*(\w*)?\]\s*\[\w+\-\w+\:\s*(\w*)?\]\s+\[([^\,]*)\]\:\s*\[([^\,]*)?\,([^\,]*)?\,([^\,]*)?\,([^\,]*)?\,([^\,]*)?\,([^\,]*)?\](.*)?

How to Start Oracle Management Cloud Agents automatically using systemctl

The Oracle Management Cloud documentation calls out how to start the OMC agents at system reboot, but at the time of writing it's light years behind still focusing on Linux 6 and /etc/init.d scripts.

If you're interested in the old school way of doing things follow the official documentation located here Configure Automatic Restart of a Cloud Agent.

If like most of the world population you've moved onto Linux 7 then you're on your own.  This is where I come in.

Hopefully I'll save you the sh*g and hassle of working out how hard this was, if you're interested comment out "RemainAfterExit=true" and then try and debug it, good luck.

Basically perform these steps as root, you'll obviously have to change User, Group and "/u01/app/omcagent/" but the rest of it should stay the same;

echo "[Unit] Description=Oracle Management Cloud Agent [Install] WantedBy=multi-user.target [Service] User=opc Group=opc LimitNOFILE=10000 LimitNPROC=10000 ExecStart=/u01/app/omcagent/agent_inst/bin/omcli start agent ExecStop=/u01/app/omcagent/agent_inst/bin/omcli stop agent ExecReload=/u01/app/omcagent/agent_inst/bin/omcli reload agent StandardOutput=syslog StandardError=syslog Restart=on-abnormal RemainAfterExit=true" > /usr/lib/systemd/system/omcagent.service systemctl enable omcagent.service systemctl start omcagent.service systemctl status omcagent.service