snmpd interference with FPP

[Setarcos]

Hi folks,

I have been using SNMP to monitor my FPP instances since starting with FPP back in 2015 and noticed an abnormality with the latest version of FPP. After upgrading one of my Pis to FPP 6.2 I noticed after installing snmpd (the standard version that is part of bullseye) that if snmpd is running, FPP's web interface is intermittently showing "Abnormal Conditions - May Cause Poor Performance
FPPD Daemon is not running", but if I have a sequence playing, the playout doesn't appear to be affected. I checked the logs and am not seeing an indication that fppd is actually stopping/restarting.

The front-end does indeed appear to be behaving properly (the /api/system/status endpoint is intermittently returning {"fppd":"Not Running","status":"stopped"... and the Abnormal Condition warning is appearing as it should), but it isn't clear why the status endpoint is actually doing this.

Any thoughts on where I should look to try to troubleshoot this further?

[JonD]

So without SNMP running you do not get the FPP abnormal message, but as soon as you install it, and setup monitoring it will appear?

How frequently is your monitor checking the status?  Are you using SNMP v1, v2, or v3?  What version of Pi are you running?

[Setarcos]

So without SNMP running you do not get the FPP abnormal message, but as soon as you install it, and setup monitoring it will appear?

The issue only appears if snmpd is running. If I manually stop the snmpd service, the FPP web interface doesn't exhibit the "Abnormal Conditions" warning issue.

How frequently is your monitor checking the status?  Are you using SNMP v1, v2, or v3?  What version of Pi are you running?

This doesn't appear to be polling interval related as FPP behaves the same if my monitoring server is polling the device, or if SNMP polling is completely disabled. When the monitoring server is running, it polls the Pi every 5 minutes.

I am currently using SNMP v2.

This is with a Pi3B.

[Setarcos]

Digging through the SystemGetStatus() function in /www/api/controllers/system.php that handles the /api/system/status endpoint responses, and it would appear that curl is timing out either connecting or waiting for a response from http://localhost:32322/fppd/status

    } else {
        // If IP[] was not provided, then status of the local machine is desired.
        //go through the new API to get the status
        // use curl so we can set a low connect timeout so if fppd isn't running we detect that quickly
        $curl = curl_init('http://localhost:32322/fppd/status');
        curl_setopt($curl, CURLOPT_FAILONERROR, true);
        curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curl, CURLOPT_CONNECTTIMEOUT_MS, 200);
        curl_setopt($curl, CURLOPT_TIMEOUT_MS, 3000);
        $request_content = curl_exec($curl);
        curl_close($curl);

        if ($request_content === false) {
            $status = exec("if ps cax | grep -q git_pull; then echo \"updating\"; else echo \"false\"; fi");

            $default_return_json['fppd'] = "Not Running";
            $default_return_json['status_name'] = $status == 'updating' ? $status : 'stopped';

            return json(finalizeStatusJson($default_return_json));
        }
        $data = json_decode($request_content, true);
        return json(finalizeStatusJson($data));
    }

I'll keep poking at this as time permits...

[JonD]

If the service only needs to be enabled, I don't have any suggestions. 

After upgrading one of my Pis to FPP 6.2 I noticed

Is this only happening on one of several Pis, or did you only install SNMP on one of them?  I could try to recreate the issue on my end if it would be helpful, but sounds like you have several Pis around for testing.

[Setarcos]

If the service only needs to be enabled, I don't have any suggestions. 

After upgrading one of my Pis to FPP 6.2 I noticed

Is this only happening on one of several Pis, or did you only install SNMP on one of them?  I could try to recreate the issue on my end if it would be helpful, but sounds like you have several Pis around for testing.

I have snmpd running on all my FPP Pis of various types (2B, zero, 3B and 4), but this is the only one I currently have running 6.x

[Setarcos]

A bit more testing on this. It seems as though the connect timeout needs to be bumped up to 3 seconds so that this doesn't happen. It isn't entirely clear why though as this seems to only be required while snmpd is running. With snmpd running if I directly load http://<my-fpp-instance>:32322/fppd/status, it is very responsive (9-32ms), but going through http://<my-fpp-instance>/api/system/status takes 209-3300ms.