Author Topic: FPPD running, playlist running but no output until FPPD restart  (Read 459 times)

Offline JSOrange97

  • Newbie
  • *
  • Join Date: Dec 2015
  • Location:
  • Posts: 27
  • Kudos: 0
FPPD running, playlist running but no output until FPPD restart
« on: January 10, 2019, 03:11:10 PM »
Running off of latest master branch on FPP on Pi3.  Status page appears to indicate playlist running as scheduled (time looks correct, etc.) but no output from any controller.  As soon as I hit "fppd restart" button, the playlist starts over and proper output via all downstream controllers.  Any ideas?

Offline CaptainMurdoch

  • Administrator
  • *****
  • Join Date: Sep 2013
  • Location: Washington
  • Posts: 9,798
  • Kudos: 208
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #1 on: January 10, 2019, 03:17:31 PM »
It sounds like the E1.31 code disabled the controllers because it couldn't talk to the briefly.  Does this happen every reboot or just this once?
-
Chris

Offline JonB256

  • Supporting Member
  • ******
  • Join Date: Mar 2013
  • Location: Granbury, Texas
  • Posts: 5,163
  • Kudos: 122
    • Granbury Christmas Lights
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #2 on: January 10, 2019, 03:20:04 PM »
Is this after an initial power on? I know there have been updates since my show ended 5 days ago, but I never had this problem. (Master branch)

Does it occur EVERY time a scheduled playlist starts or just on Power up/Boot?  (I didn't power down my players - they ran for 40 days straight but they did get frequent FPPD restarts because I was running Master branch and updated when I noticed a release.

Offline JSOrange97

  • Newbie
  • *
  • Join Date: Dec 2015
  • Location:
  • Posts: 27
  • Kudos: 0
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #3 on: January 10, 2019, 03:24:31 PM »
To answer both questions above - this happens only on initial power up.  I don't believe I have my timers staggered, so disabling E1.31 seems to make sense given what I am seeing.

Offline JSOrange97

  • Newbie
  • *
  • Join Date: Dec 2015
  • Location:
  • Posts: 27
  • Kudos: 0
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #4 on: January 10, 2019, 03:30:25 PM »
Does it occur EVERY time a scheduled playlist starts or just on Power up/Boot?  (I didn't power down my players - they ran for 40 days straight but they did get frequent FPPD restarts because I was running Master branch and updated when I noticed a release.
I suppose to add more detail to my setup - I shutdown and power everything off every night via timers.  Every day, the timers kick on and power up.  So what you both have described, I may be bringing up FPP too early, before the downstream controllers are powered up. 

I could confirm by shutting everything down and not turning on the controllers.  Allow the Pi to boot and wait to see if the scheduler starts, then power up the downstream controllers.  Then do the reverse (controllers first, then Pi.)

Offline JSOrange97

  • Newbie
  • *
  • Join Date: Dec 2015
  • Location:
  • Posts: 27
  • Kudos: 0
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #5 on: January 10, 2019, 04:04:59 PM »
It sounds like the E1.31 code disabled the controllers because it couldn't talk to the briefly.  Does this happen every reboot or just this once?


I ran a test because I thought it only happened on initial power up.  Here's what happened:
1) Started with sequence running with light output.  Turned off the downstream controllers and rebooted FPP.  On reboot, confirmed the playlist was "running".  Turned on controllers and  no light output.
2) From Step 1 - E1.31 controllers powered on, rebooted pi.  Playlist "running" but no output.  Restarted fppd and got light output


Does FPP save state from its last shutdown? Does it think there are no controllers and it is disabling output, even though the controllers are indeed powered on?

Offline CaptainMurdoch

  • Administrator
  • *****
  • Join Date: Sep 2013
  • Location: Washington
  • Posts: 9,798
  • Kudos: 208
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #6 on: January 10, 2019, 04:18:41 PM »
FPP sends out blanking data when it starts, so it may be detecting the controllers down due to that send taking too long.

Offline JSOrange97

  • Newbie
  • *
  • Join Date: Dec 2015
  • Location:
  • Posts: 27
  • Kudos: 0
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #7 on: January 10, 2019, 04:33:17 PM »
FPP sends out blanking data when it starts, so it may be detecting the controllers down due to that send taking too long.
Not sure that explains the behavior of the send taking "too long" if everything else was powered up and ready prior to FPP coming online.  Also, is FPP waiting for a return acknowledgement?

Offline JonB256

  • Supporting Member
  • ******
  • Join Date: Mar 2013
  • Location: Granbury, Texas
  • Posts: 5,163
  • Kudos: 122
    • Granbury Christmas Lights
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #8 on: January 10, 2019, 09:16:34 PM »
I understand (but don't, myself) the reason for removing power to E1.31 controllers, but I really don't recommend starting and stopping RasPi or BBB based controllers. Check them via web browser occasionally and maybe reboot if you think it needs it, but leave it powered.

Offline JSOrange97

  • Newbie
  • *
  • Join Date: Dec 2015
  • Location:
  • Posts: 27
  • Kudos: 0
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #9 on: January 11, 2019, 02:51:46 PM »
I understand (but don't, myself) the reason for removing power to E1.31 controllers, but I really don't recommend starting and stopping RasPi or BBB based controllers. Check them via web browser occasionally and maybe reboot if you think it needs it, but leave it powered.
I understand your point, but I look at it this way: if something isn't going to be used for 16 hours a day, and only needs to be on a known schedule, only turn it on when it needs to be on.  I have my Pi, Ethernet router/switch, and radio transmitter all on the same timer.  The E1.31 controllers are on a separate timer.  I can look at separating it all so the Pi is always running, but I have been using this setup for 3 seasons now.  Something has changed in 2.x in that what used to work fine for two years now doesn't work.

Offline CaptainMurdoch

  • Administrator
  • *****
  • Join Date: Sep 2013
  • Location: Washington
  • Posts: 9,798
  • Kudos: 208
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #10 on: January 11, 2019, 03:08:31 PM »
Something has changed in 2.x in that what used to work fine for two years now doesn't work.

FPP used to keep trying to send to your E1.31 controllers even if they weren't on the network.  This would in turn bog down on the network side of the OS because FPP was trying to send hundreds of packets per second to a non-existent host which in turn caused the rest of things to back up and the lights wouldn't blink properly anymore.  In FPP v2.x, Dan updated the E1.31/DDP/Artnet code to detect if a controller goes offline and stop sending data to it.  Every so often, FPP will re-check to see if the controller is back online and resume sending it data.  If your controllers aren't being marked as 'online' after powering up after fppd starts, then that is another issue that we need to look at, but the reason for the change was to solve a problem that quite a few have experienced, even when testing, when they were sending E1.31 to a non-existent (or down) controller.

Offline pixelpuppy

  • Hero Member
  • *****
  • Join Date: Aug 2015
  • Location: Dallas, TX
  • Posts: 1,360
  • Kudos: 43
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #11 on: January 11, 2019, 03:13:11 PM »
I look at it this way: if something isn't going to be used for 16 hours a day, and only needs to be on a known schedule, only turn it on when it needs to be on.  I have my Pi, Ethernet router/switch, and radio transmitter all on the same timer.  The E1.31 controllers are on a separate timer.  I can look at separating it all so the Pi is always running, but I have been using this setup for 3 seasons now.  Something has changed in 2.x in that what used to work fine for two years now doesn't work.


I do the same thing and have had no issues.  I think what has changed in v2 is the the blacklisting of offline controllers.  This is actually a performance enhancement since defined unicast controllers that are offline can cause delays if FPP keeps trying to send data to them.  The other thing is FPP v2.x boots a lot faster now.  But I'm still not seeing the problems you are. 


So the question seems to be, "why does FPP think your controllers are not online when it starts up?".  You say they are on different timers, so do you have them set so the controllers come on before the FPP?   Mine are all on the same timer so they all come on at once.  But I also turn them on at least an hour before showtime.  Since FPP boots so fast now, it may be possible that I'm getting some blacklisted controllers too but FPP keeps rechecking every few minutes and will clear them from the blacklist when it detects a controller is back online.  But it does take a few minutes for this to happen.   How much time do you allow between power up and when the show starts?
-Mark

Offline JSOrange97

  • Newbie
  • *
  • Join Date: Dec 2015
  • Location:
  • Posts: 27
  • Kudos: 0
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #12 on: January 11, 2019, 03:15:35 PM »
Something has changed in 2.x in that what used to work fine for two years now doesn't work.

FPP used to keep trying to send to your E1.31 controllers even if they weren't on the network.  This would in turn bog down on the network side of the OS because FPP was trying to send hundreds of packets per second to a non-existent host which in turn caused the rest of things to back up and the lights wouldn't blink properly anymore.  In FPP v2.x, Dan updated the E1.31/DDP/Artnet code to detect if a controller goes offline and stop sending data to it.  Every so often, FPP will re-check to see if the controller is back online and resume sending it data.  If your controllers aren't being marked as 'online' after powering up after fppd starts, then that is another issue that we need to look at, but the reason for the change was to solve a problem that quite a few have experienced, even when testing, when they were sending E1.31 to a non-existent (or down) controller.


This makes sense and now that I know about it I can try and work with it.  The "every so often" may need to be understood because on days that I noticed that I needed to do an fppd restart, it would be at least 5 minutes into the show. 

Offline JSOrange97

  • Newbie
  • *
  • Join Date: Dec 2015
  • Location:
  • Posts: 27
  • Kudos: 0
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #13 on: January 11, 2019, 03:18:56 PM »
I do the same thing and have had no issues.  I think what has changed in v2 is the the blacklisting of offline controllers.  This is actually a performance enhancement since defined unicast controllers that are offline can cause delays if FPP keeps trying to send data to them.  The other thing is FPP v2.x boots a lot faster now.  But I'm still not seeing the problems you are. 


So the question seems to be, "why does FPP think your controllers are not online when it starts up?".  You say they are on different timers, so do you have them set so the controllers come on before the FPP?   Mine are all on the same timer so they all come on at once.  But I also turn them on at least an hour before showtime.  Since FPP boots so fast now, it may be possible that I'm getting some blacklisted controllers too but FPP keeps rechecking every few minutes and will clear them from the blacklist when it detects a controller is back online.  But it does take a few minutes for this to happen.   How much time do you allow between power up and when the show starts?
The need for improvement totally makes sense.  I would turn everything on at 4:50pm for a 5:00pm show start.  Maybe that isn't enough time to recheck the blacklisted controllers.  The easy fix is to turn the controllers on at 4:45 and then the pi at 4:50.  This amount of time shouldn't be needed but if I am home I can check to see that everything is online and ready and can debug in the few minutes before show start.

Offline CaptainMurdoch

  • Administrator
  • *****
  • Join Date: Sep 2013
  • Location: Washington
  • Posts: 9,798
  • Kudos: 208
Re: FPPD running, playlist running but no output until FPPD restart
« Reply #14 on: January 11, 2019, 03:23:46 PM »
This makes sense and now that I know about it I can try and work with it.  The "every so often" may need to be understood because on days that I noticed that I needed to do an fppd restart, it would be at least 5 minutes into the show.

The 'every so often' is every 10 seconds.  I couldn't remember if it was 30 or 10, but just checked the code in UDPOutput.cpp.   If you are seeing the lights not working for 5 minutes, then that is something we need to look into.  There should be some warnings in the fppd.log file when FPP disables outputs.  Can you take a look and see if you see any?  You will need to have the 'Channel Output' Log Mask checkbox checked on the FPP Settings page or else the entries won't show.  If you see those, can you clear the fppd.log file before shutting down sometime and then after you are booted back up and the problem is occurring, grab the logs zip and post it here and we can take a look to see what fppd logged when it started up and as the playlist started playing.

 

Back to top