Health Daemon Overview

Overview

The purpose of this daemon is to monitor the health of all of the other services and translate them into signals via 3 physical LED lights. There are 3 lights this will control:

• Temp:
  • Off - No power, completely disconnected
  • Red - Daemon reporting an error
  • Orange, Solid - Daemon reporting a warning
  • Orange, slow blinking - No temp sensors connected
  • Orange, rapid blinking - Some temp sensors connected
  • Green - Temp daemon reporting temps across all sensors
• Control:
  • Off - No power, completely disconnected
  • Red - Daemon reporting an error
  • Orange - Daemon reporting a warning
  • Green - Control daemon working properly
• Web:
  • Off - No power, completely disconnected
  • Red - Daemon reporting an error
  • Warning - Daemon reporting a warning
  • Green - Web interface up and working properly

Health for each service should be monitored in the following ways:

• Temp - Read Redis health state and subscribe to temp events
  • if there are no temp events within the last 10 seconds, set LED to red
  • if there is an error reported, set LED to red
  • if there is a warning reported, set LED to orange
  • use the sensor-connected logic above in the case that any sensors are not listed
  • otherwise, set LED to green
• Control - Read health:control from Redis (optionally subscribe to health.control for faster updates). Evaluate on a short poll loop (same wait-with-timeout pattern as Control is fine when this daemon is built).
  • Payload from Control Daemon: status is ok, degraded, or error; ts is UTC ISO-8601 when the heartbeat was written.
  • status: ok → green LED
  • status: degraded → orange LED (e.g. stale temps while maintain, latched heat error phase)
  • status: error → red LED (e.g. GPIO failure, Redis publish failure)
  • Stale heartbeat: if ts is older than 10 seconds, orange LED (overrides a stale ok in the key — roughly two missed Control heartbeats at default heartbeat_s 5 s)
  • Very stale: if ts is older than 60 seconds, red LED
  • Health Daemon only drives LEDs; it does not publish events that Control waits on for safety. Stale temperature cutoff is enforced inside Control’s own 1 s reconcile loop.
  • Redis down: Control keeps reconciling locally but cannot refresh health:control → Control LED goes orange then red by age (expected).
  • Control restart: brief stale window (orange possible) until Control sends READY=1 and publishes health again.
• Web
  • Http request to API health endpoint every 5 seconds
    • Healthy - Green LED
    • Warning reported - Orange LED
    • Error reported - Red LED
  • Http request to index every 5 seconds
    • On 2xx status - Green LED
    • On error status - Red LED
    • on response lasting longer than 1 second - Orange LED

Hardware Setup

The LEDs are 3 color LEDs, Red, Green and Blue. To create red, orange and green we will be using some combination of red and green. Blue is never used. To make the colors here are the combinations:

• Red
  • Red Channel ON
  • Green Channel OFF
• Orange
  • Red Channel ON
  • Green Channel ON
• Green
  • Red Channel OFF
  • Green Channel ON

Here is the configuration for each LED:

• Temp:
  • Red - GPIO 27
  • Green - GPIO 18
• Control:
  • Red - GPIO 25
  • Green - GPIO 24
• Web:
  • Red - GPIO 6
  • Green - GPIO 12

Hardware smoke test

From the repo src directory on the Pi (stop any health daemon first):

python3 health/led_test.py
python3 health/led_test.py --dry-run   # print sequence only

led_test.py loops Temp → Control → Web. For each LED it shows red, orange, and green for one second each, then turns that LED off before moving to the next. Ctrl+C turns all channels off.