New Pi HAT board and software

[lrhorer]

Have you also considered when the Pi is intentionally SHUTDOWN (as some people do every night)?  If the Arduino doesn't recognize this shutdown, wouldn't it then power/cycle, thus waking up the Pi?

The answer to both questions is, "Yes".  The default would be for the Arduino to restart the Pi whenever it detects a shutdown regardless of the cause.  Anticipating this might not in some cases be the desired behavior, however, I am including a means to tell the Arduino to stop monitoring the Pi.  This is peripheral to the discussion above, and per request, I think I am also going to provide a means to exit this state, as well.  Thus, when one wishes to perform some sort of maintenance, one either puts the physical jumper in place that disables the Arduino in hardware, or one issues a command that has the same essential effect.  After the maintenance is complete, the jumper is removed or the alternate command is issued, thus re-enabling the Arduino function.

Seems now that it would be a good idea to have a method to easily reprogram the Arduino for software updates (as someone suggested earlier).

It is a consideration.  Of course, the Arduino has its own USB port, and there is nothing to prevent any user from doing whatever is desired via that port.  I have thought about the possibility of programming the Arduino via the SPI ports.  At this point, I do not know if that is even possible, but when I get the time, I will look into it.  At this point, although the Arduino is programmable, it is acting more as a hardware device.  It merely monitors the activity of the Pi and shuts it down if it sees none.

Would the Arduino program also be specific to FPP updates (ie could any future FPP updates impact how the Arduino operates)?

At this point, no, but of course any future update - or change by the user, for that mattter - that changes or inhibits the GPIO pin(s) used by the board would impact the  board.  That very fact suggests I might be well served to limit the number of GPIO pins used by the board to three (including the fan control and the cooler control), but that complicates the coding on both devices.  Even the use of only 1 GPIO does not guarantee it won't get used in the future by FPP, and the ability to control the fan and the external cooler adds two more.  I guess I could consider controlling via SPI, which would limit the exposure, but increase the complexity of the hardware and thus the cost.

Overall, seems this type of feature may introduce more complexity (especially burdensome trying to diagnose if other types of glitches start occurring) than solving any substantial problem.

To me, that statement is wholly unfathomable.  For nearly ten years, it has been THE problem for me with FPP.  Dealing with continuous intermittent lockups has been a monumental issue for me.  In earlier years, concomitent corruption of the SSD card made the issue truly horrendous.  That seems to have been solved, but lockups are still frequent.  Indeed, I discovered a few days ago that apparently the FPP system tends to shut down every few hours or so.  I am investigating, but in any case it seems perhaps every once in a while the restart fails.  The system I am currently monitoring last restarted by itself 14 minutes ago.  I am working on some software that will keep a log on an external system to see just how often it happens on that and other systems.

There have been other issues in the past.  The most annoying of them was the so-called "robot noise" which affected those of us with 212 FM transmitters. (Thank goodness that one seems to be fixed!) That issue was nothing, NOTHING, compared to this one.  I truly do not understand how people can downplay this problem.  I admit the fact I am disabled and cannot easily physically access my hosts makes it worse for me, personally, but still, I consider any system that intermittently fails to be quite problematical.

And as far as the trigger for a fan:  doesn't that already exist natively in the FPP?  I know I can already see each Pi CPU temp (through the FPP interface).  Isn't that info already available to simply trigger a Pi pin (to activates a fan)?

A trigger and the ability to power something are two different things.  A "standard" approach would be to employ a 4 pin fan, which would allow the Pi to both control the fan speed and to control it.  This requires two GPIO pins.  One can control such a fan, however, without monitoring it, freeing up one pin.  Power is a different matter.  The GPIO pins are not able to power a fan. A 2 pin, 5V fan can be driven from the 5V rail, or some low speed fans can even be driven from the 3.3V supply.  I do not recommend either one.  A 12V fan is what I recommend, and is included with this board.  The board also controls it.  At this point, I am using a GPIO pin off the Pi, but I think I am going to reconsider that.  If I give the Arduino more hardware responsibilities, I can limit the number of GPIO pins on the Pi to one, or maybe even zero.  Let me consider some more.

I've never had temp concerns, so it's nothing I've ever looked into—so maybe I'm missing something...

You obviously do not live in South Texas (or Arizona, New Mexico, ...) and use you system for, say, July 4th or other summer festivities.  Right now it is not an issue.  It has been unseasonably cold here, so right now the test system CPUis only sitting at 39C (102F), but wait until the outside temperature soars by over 30C (54F) this summer!  Last year the temperature inside the case exceeded 70C.  Who knows how high the CPU went?

[k6ccc]

That seems to have been solved, but lockups are still frequent. 

I don't know what you're doing, but in running two FPP instances (one for my P5 and the other for a P10) for six years, I have NEVER had FPP lockup - other than some updates required a power cycle.  Granted, my FPP instances are not doing a lot since all they are doing is taking in E1.31 data and driving a matrix.  They get powered up in mid November and stay powered for the entire show season.  During the off season, they are normally powered off, but once in a while get powered up to test something (and often get firmware updated).

[lrhorer]

That seems to have been solved, but lockups are still frequent.

I don't know what you're doing, but in running two FPP instances (one for my P5 and the other for a P10) for six years, I have NEVER had FPP lockup

I have seen others here say that, but it is most certainly not my experience.  I have had hundreds of lock-ups across more than 20 different systems running on everything from Raspberry Pi B Version 1 boards to P:i 4 boards to P:i Zero W boards.  The one under direct test right now locked up less than 2 weeks ago and automatically restarted itself successfully just under 4 hours ago.  It also locked up less than a month ago when I upgraded the OS software using apt, which sometimes happens.

Granted, my FPP instances are not doing a lot since all they are doing is taking in E1.31 data and driving a matrix.

The Pi in question is a Raspberry Pi Zero W version I, and all it is doing is running a single 200 pixel WS2811 string off pin 12.  It is scheduled to run and all-black sequence between midnight and 6:30, adn a very simple Red / Green static sequence between 6:30 and midnight.  The only software running continuously other than FPP (including NTP, DHCP etc.) is Wireguard, which is required in order to access the system.  None of the other FPP systems, which have all suffered the same issue have run Wireguard, and none of my other Wireguard clients have ever had a reboot issue.
 

They get powered up in mid November and stay powered for the entire show season.  During the off season, they are normally powered off, but once in a while get powered up to test something (and often get firmware updated).

Some of mine are powered year-round, but even the ones powered up in October (for Halloween) and shut down in February have experienced the issue time and again.

[Poporacer]

I have seen others here say that, but it is most certainly not my experience.  I have had hundreds of lock-ups across more than 20 different systems running on everything from Raspberry Pi B Version 1 boards to P:i 4 boards to P:i Zero W boards. 

I have 10 FPP devices running non-stop all year long and have not experienced this and have another 6 that are running commercial shows sending data to over 200 controllers and have yet to have a single lockup in the last month...What version of FPP and OS? I have a feeling it is a configuration issue, with almost 15,000 FPP users right now, I am sure if locking up was a common issue we would have heard more about it by now. 

The one under direct test right now locked up less than 2 weeks ago and automatically restarted itself successfully just under 4 hours ago. 

Have you posted the issue with logs in GitHub?

It also locked up less than a month ago when I upgraded the OS software using apt, which sometimes happens.

I am not sure if apt will do an os upgrade....so your OS might be out of date.

[Bwinter]

That seems to have been solved, but lockups are still frequent.

I don't know what you're doing, but in running two FPP instances (one for my P5 and the other for a P10) for six years, I have NEVER had FPP lockup - other than some updates required a power cycle.  Granted, my FPP instances are not doing a lot since all they are doing is taking in E1.31 data and driving a matrix.  They get powered up in mid November and stay powered for the entire show season.  During the off season, they are normally powered off, but once in a while get powered up to test something (and often get firmware updated).

I'm in the same boat. I've been running 7-10 Pi's (zeros, 3B, 3B+) for the last 5+ years and have never had a lockup.  I power down every night (technically, I just kill the power and don't even bother with the FPP shutdown), so they do get a fresh restart everyday.

But seeing this type of error so frequently suggests that something else is going on.

[Bwinter]

IWho knows how high the CPU went?

Did you check the FPP interface?  It seems to give a pretty reliable temperature reading.

[lrhorer]

I have 10 FPP devices running non-stop all year long and have not experienced this and have another 6 that are running commercial shows sending data to over 200 controllers and have yet to have a single lockup in the last month...What version of FPP and OS? I have a feeling it is a configuration issue, with almost 15,000 FPP users right now, I am sure if locking up was a common issue we would have heard more about it by now.

One might think, perhaps, but then there are a lot of corners in this arena.  Other than using a V-FMT212R and handling pixel strings, I can't think of what corners I might be in.  Even so, both of those are modestly large corners, I think, and none of my systems do both.  I don't think I can claim all of my systems have experienced it, because some are only used for a few days in the year (Easter, 4th of July, Thanksgiving, etc), and I don't recall specifically any lock-ups during those festivals.  It's been pretty rare that I had more than 1 lockup in a week, if ever.

Have you posted the issue with logs in GitHub?

No, at least not recently.  I may have done in the past.  I have posted so many issues to GitHub over the years - not just for this platform, but many others, as well - there is no way I would be able to remember them all.

I don't know the logs will be much good.  I don't see anything that seems to be related to the lockup.  There are quite a few different rather minor errors, but the logs just stop on the 15th and start again on the 20th.  There is some oddness with the timestamps in the Apache logs.  I think I will hold off on posting in GitHub until 7.4+ is installed on the systems, or at least most of them, in January.  I will see then if the problem persists.  Meanwhile, I am going to fork this thread off into a troubleshooting thread.

I am not sure if apt will do an os upgrade....so your OS might be out of date.

Uh... what?  Apt upgrades every .deb package when requested, and the .deb packages represent every single bit of the OS and then some, with the exception of custom compiled drivers and binaries.  Now, 5.5 is running on Buster, so by definition the OS is out of date, I suppose, but then so is 5.5.  Presumably the entirety of the FPP package is in /opt/FPP.  If not, I need to have a little chat with the developers.  Looking at /opt/FPP/src, there are a fair number of drivers and some ordinary binaries, but I don't see anything that would conflict with anything in the OS or loaded software other than ping and GPIO.  Ping seems to work just fine and I don't have GPIO loaded, so I don't see any conflicts.  The developers here have been quite good about most conflicts, with a few exceptions, in the past.  I suppose it's entirely possible, but the failures long predate this OS upgrade.  It was only done to allow me to insert my own scripts for monitoring.

[Poporacer]

If not, I need to have a little chat with the developers. 

You better chat with the developers then!

Now, 5.5 is running on Buster

So all of this "problem solving" you are trying to fix is based on versions that are almost 2 years old and potentially not even a fully functional operating system?

[lrhorer]

IWho knows how high the CPU went?

Did you check the FPP interface?  It seems to give a pretty reliable temperature reading.

Since the system locked up - in this case presumably due to heat - no, I did not.  Honestly, despite the Pis ability to throttle its CPU when CPU temperatures exceed 80C (85C?), I really do not want any system not water, oil, or liquid metal cooled to be running in environmental temperatures higher than 40C.  Whenever an air cooled case has a higher internal temperature than 40C, I want to shut it down.  So, IMO, should anyone else.  In many of the Southern states and places like Africa, Southern Asia, Australia, etc., such temperatures when fully exposed to the Sun are pretty much inevitable in the Summer, and even sometimes in Winter.  The addition of a Peltier cooler is, I think, both a very good idea and quite economical.  Amazon has them for around $6, or with a 12V fan and fancy heat sink / mounting system, around $20.  Both the Version I and Version II boards I am developing can drive either one.  (Version II supports two fused pixel strings of up to 7A 12/24V each, and more sensors.)

[k6ccc]

It also locked up less than a month ago when I upgraded the OS software using apt, which sometimes happens.

Why are you using apt to upgrade?  The way FPP is setup, should never require apt for anything.

[lrhorer]

You better chat with the developers then!

Why?  Do you know of something the developers have done outside the src directory?  Such would be exceptionally bad form, almost insane.  I have been working with computers for quite some time, and I have never seen any software whose source code was outside the Makefile's directory.  Indeed, I haven't looked at all 19 Makefiles in the FPP archive, but since /opt/fpp/src/Makefile contains the line `SRCDIR = /opt/fpp/src`, I feel pretty confident in the assumption for the time being.  There are also 177 driver files under /opt/fpp, and I checked but none are duplicated in /usr/lib.  According to lsmod, none of them are loaded by the OS, so, no I don't think anything foolish has been done.

Now, 5.5 is running on Buster

So all of this "problem solving" you are trying to fix is based on versions that are almost 2 years old[/quote]
Up to 10 years, more likely, since the issue has been there ever since I first loaded FPP.  So what?  Oh, and since I do not in practice ever upgrade any deployed systems between September and the middle of January, it's more like 1 year, but still, so what?  Until I verify it is no longer an issue in 7+, I am not going to assume it is not.  Guess what, though?  Even if 7+ no longer has the problem, or perhaps I can alleviate it, I am not going to dump these designs.  They do far too many things I consider essential.

and potentially not even a fully functional operating system?

There is virtually no chance whatsoever the OS is not 100% functional to the extent Buster ever was.  No offense intended, but you seem not to even know what apt is or does, your status as a developer notwithstanding, so such comments in this respect would seem at best superfluous.

[lrhorer]

It also locked up less than a month ago when I upgraded the OS software using apt, which sometimes happens.

Why are you using apt to upgrade?  The way FPP is setup, should never require apt for anything.

While this is perhaps true for typical users, it is not true for developers, like me.  Specifically, I need some Python utilities that were not properly supported when I loaded FPP 5.5.  If properly written, which to my knowledge FPP has mostly been, upgrading the OS should not usually (ever, really) badly impact any custom compiled binaries, the exception being shared  environment variables.  Looking at 5.5, which I have done, this seems true.  Not only that, but any moderately advanced user (like me, as well), may well want to make use of any one of the tens of thousands of applications available to any Debian-based distro.  I think a number of users, and I expect developers as well, run FPP under Debian or Ubuntu directly on desktop or server machines.  After all, why not?  I don't do it myself, but what is wrong with driving some pixels directly off one's desktop machine or a server?  For that matter, I am not sure why one of the FPP developers does not offer FPP as an unofficial Debian package.

More to the point, whether one uses apt or FPP's built-in OS upgrade, lockups during an OS upgrade happen.  I have seen it quite literally thousands of times.  Indeed, whenever I upgrade any OS, I pretty much expect a lockup or two.  Note I did not upgrade beyond Buster, which is the OS and Kernel level for which 5.5 was developed.  That would be asking for trouble.

[Bwinter]

IWho knows how high the CPU went?

Did you check the FPP interface?  It seems to give a pretty reliable temperature reading.

Since the system locked up - in this case presumably due to heat - no, I did not.  Honestly, despite the Pis ability to throttle its CPU when CPU temperatures exceed 80C (85C?), I really do not want any system not water, oil, or liquid metal cooled to be running in environmental temperatures higher than 40C.  Whenever an air cooled case has a higher internal temperature than 40C, I want to shut it down.  So, IMO, should anyone else.  In many of the Southern states and places like Africa, Southern Asia, Australia, etc., such temperatures when fully exposed to the Sun are pretty much inevitable in the Summer, and even sometimes in Winter.  The addition of a Peltier cooler is, I think, both a very good idea and quite economical.  Amazon has them for around $6, or with a 12V fan and fancy heat sink / mounting system, around $20.  Both the Version I and Version II boards I am developing can drive either one.  (Version II supports two fused pixel strings of up to 7A 12/24V each, and more sensors.)

Sorry, but I'm not concerned AT ALL if the CPU is operating at 40C.  That is FAR below the recommended operating temperature.

[lrhorer]

Here is the fork for the lockup issues

Here is the Fork for hardware development.

I will continue to address the software development for the boards in this thread.

[lrhorer]

Sorry, but I'm not concerned AT ALL if the CPU is operating at 40C.  That is FAR below the recommended operating temperature.

That is not at all what I said.  'Not even close.  Please read my post again.  If you still have questions, please ask.