Intermittent fpp hangs (7.2.x, 7.3.5)

[aggieNick02]

My setup was solid last year, but upgrading to 7.2 then 7.3 this year I am getting an intermittent hang every 4 days or so. The last two times the web UI was not responsive and the show had frozen - unfortunately each time I had to reboot without trying to investigate more via ssh.

Looking at the logs for both hangs, each hang had the last (non-noise) line in /var/log/syslog:

FPP systemd[1]: Starting phpsessionclean.service - Clean php session files...

(The log is also filled with constant "pcf85363_rtc_read_time: error -121" errors every 2 seconds, but those just seem like log noise.)

I just updated to 7.3.6 right now - the last hang was on 7.3.5 tonight.

Any ideas?

[jtrpop]

Just happened to me. Viewer IMed me saying no sound with lagging lights. When I got to check it FPP was inaccessible and lights all dark. Rebooted it and it came back up. Never had these issues before. On 7.3.5 and see 7.3.6 was released. Will need to update after show is done.

[aggieNick02]

Interesting. Any idea how many days your FPP was running before it went down? The one change in 7.3.6 that caught my eye was fixing lots of memory leaks. I installed atop so if it happens again I can see if memory was climbing to exhaustion before it froze.

[CaptainMurdoch]

Do you know if you could ssh or was it totally offline? 

[jtrpop]

@CaptainMurdoch When I realized my show was dark, I tried to SSH but could not. Inaccessible via web too. I don't recall if I could ping it or not. Pulled power, plugged in, it came back up.

It had been running a few days on 7.3.5 without issue, but last week (older FPP, maybe 7.3.4?) I did have some laggy lights once, audio was fine though. Reboot fixed it. I'm using exact same hardware and sequences from last year. Only change was two new Sandisk SSDs (one for my Pi master, the other my Pi remote) flashed fresh with 7, upgraded to 7.3.5, backup imported from a FPP backup. So really only change is FPP version and the new SD cards. Have never had these issues before.

Guy IMed me and told me what sequence it started glitching out on if I could find it in the logs if I know what to look for.

Thanks

[jtrpop]

Upgraded to 7.3.6. Let's see how that goes  

[aggieNick02]

Do you know if you could ssh or was it totally offline? 

Unfortunately I don't know. First of the two times I was leaving the house and only had time to unplug and replug, and last night my son did the same as I had just got in the shower when he noticed it. If it happens again, I'll see if I can ssh or get ping responses.

The log from last night continued after the hang until power reset (nest cam audio let me discern when it happened). But after the hang the log only had the read_time noise entries. So at least that time, the system was not completely hung.

Happy to share any logs, etc, that might help

[jtrpop]

7.3.6 and it ran for a few days, but then borked again tonight. Lights frozen at one spot, no audio. I could SSH, but it did seem poky. Web GUI was accessible, but said "FPPD Daemon is not running" and reboot did nothing. Had to pull power to reset the Pi. I'll try 7.4 next, might swap the SD (which is brand new), but I'm hearing lots of other people with similar issues with 7.x in different Facebook groups.

[CaptainMurdoch]

7.3.5 and prior had some memory leaks which have now been fixed.  If you can still reproduce in 7.4, then we'll need to take a look at the fppd.log or crashlog.

[falknerlights]

As others have posted, I have had several FPP lockups or crashes since upgrading to 7.x. I had no lockups last year. I have already had one instance of it occurring in 7.4 (last night). Could get to the web browser and it said FPPD daemon not running. Restarting FPP did not solve it, had to reboot the Pi via the web browser. My system has one master Pi driving sACN to the rig and two remote Pi's with P5 panels. Only lockups appear to have been on the master. Also noted that after the reboot only the last scheduled event ran (I have an event that runs a few minutes before the main show to run a script to take control from a Mosaic MSC running permanent lights, which did not run after the reboot). The master Pi is kept powered up 24/7 and is on a UPS, while the remotes are powered down each night.

[Poporacer]

I have already had one instance of it occurring in 7.4 (last night). Could get to the web browser and it said FPPD daemon not running.

How about the log files to give the developers something meaningful to look at? And the date and time that this happened.

[falknerlights]

Which logs are they looking for, and what is the best way to get them to them?

[Poporacer]

Go to Content Setup>File Manager and then the Logs tab. Click on the Zip button...It will download all the log files.

[falknerlights]

Attached. I updated to 7.4 on 12/9 and the first time the show didn't run under 7.4 was 12/11 (had to reboot Pi). On 12/12 the take control event did not run, but the main show did. I am also getting a lot of can't ping controller messages during the show, but not lagging or loss of control was observed on the display.You cannot view this attachment.

[Poporacer]

Attached.

Thanks...a few things. I see a TON of network errors with the controllers coming on and off line which could be some sort of networking issue. I would look at that!
Also there are some errors with the FPPMon plugin that looks like it is network related as well? I would remove the plugin until you get the networking resolved.

You are also getting an error on the  White Christmas - 1s Leadin.mp3 file. I would suggest deleting this file. Re-encoding the original and then re-uploading it to make sure you didn't get a corrupt file somehow