Source randomly getting removed all of a sudden?

  • This is a bit of a weird one. I've had 2.0.0-alpha.9 up and running for a long time controlled by HomeAssistant automations (using Kodi and time triggers) and it's been working flawlessly.


    Now after moving I ran into issues, without changing any software, where calls from HA automations only work about 50% of the time. I've updated to 2.0.13 and latest HA but the same issue remains.


    The problem seems to be that priority handling is very inconsistent and the HA source randomly getting removed for no apparent reason. For example I have an automation that turns screen capture on when starting playback in Kodi and then turns on an effect on stop. This is what happens when starting playback:


    Code
    2022-09-20T20:16:50.582Z [MUXER|First LED Hardware instance] (DEBUG) (PriorityMuxer.cpp:359:updatePriorities()) Removed source priority 128
    2022-09-20T20:16:50.582Z [MUXER|First LED Hardware instance] (DEBUG) (PriorityMuxer.cpp:422:updatePriorities()) Set visible priority to 250
    2022-09-20T20:16:50.583Z [SMOOTHING|First LED Hardware instance] (DEBUG) (LinearColorSmoothing.cpp:683:selectConfig()) [0] - type: Linear, pause: false, settlingTime: 100ms, interval: 25ms (40Hz), delay: 0 frames
    2022-09-20T20:16:50.792Z [EFFECTENGINE|First LED Hardware instance] (INFO) Effect [Sea waves] finished


    Source priority 128 is set in HomeAssistant (in this case Sea waves was running before starting playback).


    On stop (when it works as intended):


    Code
    2022-09-20T20:14:12.362Z [EFFECTENGINE|First LED Hardware instance] (INFO) Run effect "Sea waves" on channel 128
    2022-09-20T20:14:12.362Z [EFFECTENGINE|First LED Hardware instance] (DEBUG) (EffectEngine.cpp:206:runEffectScript()) Start the effect: name [Sea waves]
    2022-09-20T20:14:12.362Z [MUXER|First LED Hardware instance] (DEBUG) (PriorityMuxer.cpp:182:registerInput()) Reuse input 'Home Assistant@::ffff:192.168.1.4/EFFECT' (Sea waves) with priority 128
    2022-09-20T20:14:12.583Z [MUXER|First LED Hardware instance] (DEBUG) (PriorityMuxer.cpp:422:updatePriorities()) Set visible priority to 128
    2022-09-20T20:14:12.584Z [SMOOTHING|First LED Hardware instance] (DEBUG) (LinearColorSmoothing.cpp:683:selectConfig()) [6] - type: Linear, pause: false, settlingTime: 200ms, interval: 40ms (25Hz), delay: 0 frames


    Here the effect is restarted again on stop and everything works as expected. But now about 50% of the time instead I get:

    Code
    2022-09-20T20:12:58.536Z [EFFECTENGINE|First LED Hardware instance] (INFO) Run effect "Sea waves" on channel 128
    2022-09-20T20:12:58.536Z [EFFECTENGINE|First LED Hardware instance] (DEBUG) (EffectEngine.cpp:206:runEffectScript()) Start the effect: name [Sea waves]
    2022-09-20T20:12:58.536Z [MUXER|First LED Hardware instance] (DEBUG) (PriorityMuxer.cpp:182:registerInput()) Reuse input 'Home Assistant@::ffff:192.168.1.4/EFFECT' (Sea waves) with priority 128
    2022-09-20T20:12:58.582Z [MUXER|First LED Hardware instance] (DEBUG) (PriorityMuxer.cpp:359:updatePriorities()) Removed source priority 128


    Here the source is immediately removed again and I have no idea why. It seems completely random.

  • osen

    Added the Label Issue
    • New
    • Official Post

    osen I think it is related to a background effect issue, which was fixed lately.


    Improve Led Device on/off and background effect by Lord-Grey · Pull Request #1502 · hyperion-project/hyperion.ng
    Summary Fixes Queue LED-device switch off and on signals address LGTM findings Update Do not switch-off LED-device, if background effect is configured an…
    github.com

  • Lord-Grey That looks promising :) How would I go about testing this fix, I'm guessing building from source? I'm using Hyperion on a RPi3B running LibreELEC which complicates things a bit and I'm not very good at these things unfortunately.


    I tried cross compiling for Buster using WSL2 and the provided Docker script but unfortunately it fails halfway through with the following error:



    Building for Bullseye worked but I don't think those files are compatible with LibreELEC 9.2.8, I will try though.

    Edited once, last by osen: Merged a post created by osen into this post. ().

    • New
    • Official Post

    Hi osen

    Try to build on e.g Ubuntu 22.04 via docker.
    Clone into a new directory and run in hyperion's root directory

    sudo ./bin/scripts/docker-compile.sh -i armv7l -t stretch -b Debug -p true -l

    You find the package in the ./deploy/armv7l/stretch directory

    To save you some time, I built you a tar:
    https://ufile.io/54rq79gx

  • Thank you very much :) That went way over my head so I just used your tar.


    Unfortunately I still run into the same issue. I just tested solid colours instead of effects due to what you thought the issue was and that seems to work without issues (i.e. properly transitions between solid colour and platform capture on playback start/stop in my case) or inconsistencies.


    I also dug deeper into the HA logs and found that it sometimes throws the following error when using effects:


    WARNING (MainThread) [hyperion.client] Failed Hyperion (192.168.1.3:19444) command: {'command': 'clear', 'error': 'Errors during specific message validation, please consult the Hyperion Log', 'success': False, 'tan': 0}


    Looking at the Remote Control section of Hyperion there are a lot of weird things going on when using effects in the automations. Sometimes there is no change at all (effect still present and active), sometimes Origin and Type fields are empty while Priority remains and source is still active (blocking platform capture due to higher priority), sometimes it's properly cleared and works as intended and sometimes LED output is turned off completely.

    • New
    • Official Post

    osen

    1.When testing with the package provided by myself, did you verify that Hyperion's version is 2.0.14-beta something?
    (via System->About)


    2. Which version of HA are you running?
    I tested with Home Assistant 2022.8.6 and it seems to work (I have no errors in both logs), but I might have tested differently than you...

  • Lord-Grey


    Yes, it's running:


    Version2.0.14-beta.1
    Buildmaster (LordGrey-525e3537/9d23d9e2-1662531133)
    Build dateSep 21 2022 18:47:16


    Right now I'm running Home Assistant Core 2022.9.6 but I had the same issue in earlier versions. I ran a super old version when this issue started alongside Hyperion 2.0.0-alpha.9 which worked fine before I moved (if it ain't broke, don't fix it and all that). I'll try downgrading to 2022.8.6 and report back.


    Followup: did not make a difference unfortunately.

    The problematic transition is from (any) effect to platform capture and back. When platform capture is triggered from HA it tries to clear the source from Hyperion and only leaves:


    SystemCapture Screen: (Dispmanx)250Select Source


    This interaction works without issues when using a solid colour but not when using effects it seems. Looks to me like it isn't cleared properly in the latter case. But this may be an issue with the HA integration rather than Hyperion, I'm not qualified to assess that.


    Looking through the source it seems that there is some workaround in place for clearing effects.

    Edited 3 times, last by osen: Merged a post created by osen into this post. ().

    • New
    • Official Post

    I do not think you need to downgrade.

    Latest changes in HA in regard to Hyperion seems to have happened already a while ago.


    I might better need to understand what is send to Hyperion as commands.

    That the clear command failed parsing is a concern.


    It could be that too many requests are issues in a short time and Hyperion cannot deal with them properly or HA creates wrong requests.


    I might build you a version where requests and responses are logged and then we see better what is actually happening.


    I only testest setting color or effects from HA manually and that worked.

  • I just tried changing manually between random different effects in HA and I run into the same issue sometimes, but it's very inconsistent:



    On the last lines the source is removed/cleared instead of changing effects. Then when trying to run a new effect in this state (after the above):



    And then again switching effects:


    Code
    2022-09-22T19:47:53.159Z [MUXER|First LED Hardware instance] (ERROR) Previous line repeats 200 times
    2022-09-22T19:47:54.655Z [MUXER|First LED Hardware instance] (ERROR) Previous line repeats 99 times
    2022-09-22T19:47:54.655Z [EFFECTENGINE|First LED Hardware instance] (INFO) Run effect "Fire" on channel 128
    2022-09-22T19:47:54.655Z [EFFECTENGINE|First LED Hardware instance] (DEBUG) (EffectEngine.cpp:206:runEffectScript()) Start the effect: name [Fire]
    2022-09-22T19:47:54.655Z [MUXER|First LED Hardware instance] (DEBUG) (PriorityMuxer.cpp:182:registerInput()) Reuse input 'Home Assistant@::ffff:192.168.1.4/EFFECT' (Fire) with priority 128
    2022-09-22T19:47:54.682Z [MUXER|First LED Hardware instance] (DEBUG) (PriorityMuxer.cpp:422:updatePriorities()) Set visible priority to 128
    2022-09-22T19:47:55.180Z [EFFECTENGINE|First LED Hardware instance] (INFO) Effect [Breath] finished


    Here the effect runs normally again.


    Thank you very much for the help so far, let me know if there's anything else I can do.

    • New
    • Official Post

    osen Hi


    I build you a version which logs the requests and responses, so we can see what is happening and how HA communicates with Hyperion:

    https://ufile.io/80pfifoj


    Best would be, if you run Hyperion from the command line, as the output is not written to the Hyperion log (to avoid truncation), but to the standard output.

  • Lord-Grey


    Certainly. Sorry, but how would I go about doing that exactly? I've installed Hyperion-ng using this script.

    I tried simply running ./hyperiond --userdata /storage/hyperion/ from the storage folder, is this the correct way?


    Followup: Here's some initial logs. First few effects fails to run here.

    Edited 2 times, last by osen: Merged a post created by osen into this post. ().

  • Lord-Grey


    Of course, here it is. I temporarily switched back to your previous tar as the latest debug version kept freezing the system, but I don't think that would affect the config, right?

    • New
    • Official Post

    Thanks for providing the log. I can somehow reproduce the situation now.
    It looks like it is a timeing issue.
    During startup, the grabber is still registering for priorities and then the clear and new effect is issued...
    That causes that the clear will kill the effect registration and therefore the effect does not kick-in.

    If you wait after the grabber is registered and then issue the commands, it works.

    I will need to investigate how this can be addressed.


    Is there an option that you wait e.g. 10 seconds before you clear and start the effect?
    ... but I do not know how your automation script looks like...

    Best would be, if there is no clear issued before starting the next effect command.
    It looks like that this is in the HA code, because of some previous problems in Hyperion (which were fixed).

    https://github.com/home-assist…ts/hyperion/light.py#L322

    Not sure, if there is an option, that you could test without that code on clear before effects..?

  • I haven't tinkered with automation in a long while but I think it should be possible to add a delay between dual actions (turn off effect -> delay -> turn on other effect) instead of just having a single action changing effect. I don't know when the clear command is sent though.


    I'm running Home Assistant Operating System, which I believe uses nestled Docker containers, making accessing the actual files complicated (especially when you're not very tech savvy like me ...). It may be possible to use custom components for this and "sideload" and edited version of the integration (never done it). I'll look into it tomorrow.

  • Didn't have time yesterday, sorry. I think it's running as a custom component with the edited light.py but I'm not 100% sure as I don't know how to verify that HA uses those files for the integration rather than the core ones. It is indicating that the integration is a custom component with it though. I simply removed the following:


    Code
    if not await self._client.async_send_clear(
                    **{const.KEY_PRIORITY: self._get_option(CONF_PRIORITY)}
                ):
                    return

    Was that correct?

    Anyway it seems to have made things even worse. Now the transition effect to platform capture never works. Here's a log.


    Adding a delay in the automation going light.turn_off -> 3s delay -> light.turn_on did not work either, log here. Using the original light.py did not change this.

  • Seeing as the transitions between solid colour and platform capture works without a hitch I tried another workaround: using a solid colour with zero brightness ("black") between the transition (effect - > 3s delay -> black -> 3s delay -> platform capture and other way around).


    This did not improve the situation sadly, still the same issue.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!