MQTT Watchdog Logger
Relevant source files
The following files were used as context for generating this wiki page:The Pulsar Dome Driver implements a multi-layered watchdog system to ensure operational reliability and provide real-time status monitoring via MQTT. This system is composed of two main parts: an ActionWatchdog for monitoring specific, long-running dome commands (e.g., slewing, opening/closing the shutter), and a system-level watchdog that monitors the health of the main polling loop.
When enabled, these watchdogs publish status updates, success/failure/timeout events, and heartbeats to an MQTT broker. This allows external systems to monitor the dome's health and react to events. The system also includes a WatchdogLogger utility for direct file logging of watchdog activities, although its use appears to be for debugging as it is commented out in the primary logic.
Sources: Helper/ActionWatchdog.cs, Driver/Pulsar_DomeDriver.cs:70-76, 615-671, MQTT/WatchDogLogger.cs
System Architecture
The watchdog functionality is not a single class but a collaboration between several components. The DomeDriver class orchestrates the watchdogs, which in turn use the MqttPublisher to send messages. The ConfigManager provides the necessary configuration flags to control their behavior.
Core Components
| Component | File | Description |
|---|---|---|
ActionWatchdog | Helper/ActionWatchdog.cs | Monitors a specific, long-running task. It checks for success, failure, or timeout and publishes results to MQTT. It can also trigger a reset routine. |
| System Watchdog | Driver/Pulsar_DomeDriver.cs | A background task that monitors the main polling loop for stalls and publishes a periodic MQTT heartbeat. |
MqttPublisher | MQTT/MqttPublisher.cs | A generic class for connecting to an MQTT broker and publishing messages. Used by both watchdogs. |
ConfigManager | Config/ConfigManager.cs | Manages all driver settings, including flags like UseMQTT and WatchdogRunning that control the watchdog and MQTT functionality. |
WatchdogLogger | MQTT/WatchDogLogger.cs | A static utility for writing watchdog-specific log messages to a text file. Its usage is currently commented out. |
Sources: Helper/ActionWatchdog.cs, Driver/Pulsar_DomeDriver.cs, MQTT/MqttPublisher.cs, Config/ConfigManager.cs, MQTT/WatchDogLogger.cs
ActionWatchdog
The ActionWatchdog class is the primary mechanism for monitoring asynchronous dome operations. It is instantiated for each command that requires monitoring, such as OpenShutter, CloseShutter, SlewToAzimuth, FindHome, and Park.
Sources: Helper/ActionWatchdog.cs, Driver/Pulsar_DomeDriver.cs:1484-1695
Class Structure
The ActionWatchdog class encapsulates the logic for monitoring a task against a timeout.
This diagram shows that ActionWatchdog depends on MqttPublisher for sending messages and ConfigManager for state and configuration.
Sources: Helper/ActionWatchdog.cs:10-200
Constructor Parameters
The ActionWatchdog is highly configurable via its constructor, allowing it to be adapted for different actions.
| Parameter | Type | Description |
|---|---|---|
config | ConfigManager | Reference to the application's configuration and state manager. |
timeout | TimeSpan | The duration to wait before the action is considered timed out. |
action | string | A short, internal name for the action (e.g., "home"). |
longAction | string | A descriptive message for the action being started (e.g., "Finding home..."). |
checkStatus | Func<WatchdogResult> | A function delegate that the watchdog calls to check the current status of the operation. |
internalCts | CancellationTokenSource | A token source to cancel the watchdog's monitoring loop. |
resetRoutine | Func<Task> | An optional delegate to a routine that is invoked on failure or timeout. |
mqttPublisher | MqttPublisher | An instance of the MQTT publisher for sending messages. |
mqttTopic | string | The base MQTT topic for publishing status updates. |
mqttSuccess | string | The message to publish on successful completion. |
mqttFail | string | The message to publish on failure. |
Sources: Helper/ActionWatchdog.cs:33-56
Operational Flow
The Start() method contains the main monitoring logic. It runs a loop that periodically calls the checkStatus function until the action succeeds, fails, times out, or is cancelled.
This flowchart illustrates the lifecycle of an ActionWatchdog instance from start to completion, including status checks, MQTT publishing, and optional reset triggers.
Sources: Helper/ActionWatchdog.cs:58-140
WatchdogResult Enum
The checkStatus function delegate must return a value from the WatchdogResult enum to inform the watchdog of the operation's state.
| Value | Description |
|---|---|
InProgress | The action is still running. |
Success | The action completed successfully. |
Failure | The action failed due to a known, non-error condition. |
Timeout | The action timed out (set internally by the watchdog). |
Error | The action encountered an error. |
Sources: Helper/ActionWatchdog.cs:24-31
System Polling Watchdog
Separate from the ActionWatchdog, the DomeDriver class implements a simpler, persistent watchdog. Its purpose is to monitor the health of the serial communication polling loop and to publish a periodic MQTT heartbeat.
This watchdog is started in the StartWatchdog() method and runs as a background Task.
Sources: Driver/Pulsar_DomeDriver.cs:615-671
Logic Flow
- The watchdog starts and enters an infinite loop, cancellable by a
CancellationToken. - It waits for an interval defined by
_systemWatchdogInterval(5 seconds). - MQTT Heartbeat: If MQTT is connected, it publishes the current UTC time and dome status to the
Dome/Watchdogtopic. - Polling Stall Detection: It checks the time elapsed since the last successful poll (
_lastPollTimestamp). If this exceeds_pollingStallThreshold(10 seconds), it assumes the polling loop is stalled. - Recovery: On a detected stall, it stops and restarts the polling loop by calling
StopPolling()andStartPolling().
This diagram shows the continuous loop of the system watchdog, checking for polling stalls and sending heartbeats.
Sources: Driver/Pulsar_DomeDriver.cs:615-671
MQTT Integration
MQTT is a key feature of the watchdog system, providing external visibility into the driver's state. The MqttPublisher class handles the connection and message publishing. The feature is enabled by the UseMQTT setting, which can be configured in the driver's setup dialog.
Sources: MQTT/MqttPublisher.cs, Config/ConfigManager.cs:220-231, UI/SettingsForm.Designer.cs:318-326
MQTT Topics
The driver defines several topics for publishing status and alarm information.
| Topic | File | Purpose |
|---|---|---|
Dome/DriverStatus | Driver/Pulsar_DomeDriver.cs:86 | General driver status messages (e.g., "Driver connected"). |
Dome/Dome/Status | Driver/Pulsar_DomeDriver.cs:87 | Status updates for dome rotation actions. |
Dome/Shutter/Status | Driver/Pulsar_DomeDriver.cs:88 | Status updates for shutter actions. |
Dome/Alarm | Driver/Pulsar_DomeDriver.cs:89 | Used for publishing critical failure and alarm messages. |
Dome/Watchdog | Driver/Pulsar_DomeDriver.cs:90 | Used by the system polling watchdog for its heartbeat. |
Sources: Driver/Pulsar_DomeDriver.cs:86-90
Interaction Sequence
The following diagram shows the sequence of events when a dome command like OpenShutter is executed with the ActionWatchdog and MQTT enabled.
This sequence illustrates how a client command initiates a process that is monitored by the ActionWatchdog, which reports its progress and final status via MQTT.
Sources: Driver/Pulsar_DomeDriver.cs:1484-1520, Helper/ActionWatchdog.cs:58-140
WatchdogLogger Utility
The project includes a simple, static WatchdogLogger class designed to write log messages to a file. It is thread-safe and writes to a daily log file in D:\OneDrive\Documents\Logs.
However, all calls to WatchdogLogger.Log() within the ActionWatchdog class are commented out, suggesting this logger is either a legacy component or used only for temporary debugging and is not part of the standard operational logging, which is handled by the FileLogger class.
Sources: MQTT/WatchDogLogger.cs:7-32, Helper/ActionWatchdog.cs:62, 126, 132, 138, 157, 163, 169, 180, 184, 188, 195
Conclusion
The MQTT Watchdog and Logger system is a robust feature of the Pulsar_DomeDriver that enhances reliability and observability. The ActionWatchdog provides granular, timeout-aware monitoring for specific hardware commands, while the system watchdog ensures the core polling mechanism remains active. By publishing detailed status and heartbeat messages to configurable MQTT topics, this system allows for sophisticated external monitoring, automation, and alerting, making the dome driver more resilient and easier to integrate into a larger observatory control system.