Action Watchdog
Relevant source files
The following files were used as context for generating this wiki page:The ActionWatchdog is a crucial component within the Pulsar_DomeDriver responsible for monitoring the execution of long-running hardware commands, such as slewing the dome or opening the shutter. Its primary purpose is to ensure that these actions complete successfully within a predefined timeout period. If an action fails, times out, or encounters an error, the watchdog can trigger a pre-configured reset routine, enhancing the driver's overall reliability and fault tolerance.
This system provides a robust mechanism to prevent the driver from getting stuck in an unresponsive state if the hardware fails to complete a command. It integrates with MQTT for real-time status reporting and relies on a shared configuration state to monitor the progress of hardware actions. The ActionWatchdog is instantiated and managed by the main DomeDriver class, which configures it for specific dome operations.
Sources: Helper/ActionWatchdog.cs, Driver/Pulsar_DomeDriver.cs
Architecture and Core Components
The ActionWatchdog is implemented in the Pulsar_DomeDriver.Helper.ActionWatchdog class. It operates by running an asynchronous monitoring loop that periodically checks the status of an ongoing hardware action.
Sources: Helper/ActionWatchdog.cs
Class Structure
The ActionWatchdog class encapsulates the logic for timing out an action, checking for success, and handling various outcomes. It is designed to be highly configurable through its constructor, which accepts delegates for status checking and reset procedures.
This class diagram shows the ActionWatchdog and its key dependencies.
Constructor Parameters
The behavior of each ActionWatchdog instance is defined at creation. The DomeDriver provides specific parameters and function delegates for each command it executes.
| Parameter | Type | Description | Source |
|---|---|---|---|
config | ConfigManager | A reference to the shared configuration and state manager. | Helper/ActionWatchdog.cs:39 |
timeout | TimeSpan | The maximum duration to wait for the action to complete. | Helper/ActionWatchdog.cs:40 |
action | string | A short name for the action, used in logging. | Helper/ActionWatchdog.cs:41 |
longAction | string | A descriptive message for the action, published to MQTT. | Helper/ActionWatchdog.cs:42 |
checkStatus | Func<WatchdogResult> | A delegate function that the watchdog calls to check the current status of the action. | Helper/ActionWatchdog.cs:43 |
internalCts | CancellationTokenSource | A token source to signal cancellation of the watchdog task. | Helper/ActionWatchdog.cs:44 |
resetRoutine | Func<Task> | An optional delegate function to be invoked on timeout or failure. | Helper/ActionWatchdog.cs:45 |
mqttPublisher | MqttPublisher | An optional MQTT client for publishing status messages. | Helper/ActionWatchdog.cs:46 |
mqttTopic | string | The base MQTT topic for status updates. | Helper/ActionWatchdog.cs:47 |
mqttSuccess | string | The message to publish on successful completion. | Helper/ActionWatchdog.cs:48 |
mqttFail | string | The message to publish on failure. | Helper/ActionWatchdog.cs:49 |
WatchdogResult Enum
The outcome of the monitoring process is represented by the WatchdogResult enum. The checkStatus delegate returns one of these values on each check.
| Value | Description |
|---|---|
InProgress | The action is still running and has not completed, failed, or timed out. |
Success | The action has completed successfully. |
Failure | The action has failed explicitly (e.g., hardware reported an error state). |
Timeout | The action did not complete within the specified timeout period. |
Error | The action resulted in an error state. |
Sources: Helper/ActionWatchdog.cs:27-35
Execution Flow
The ActionWatchdog is initiated by the DomeDriver when a command is sent to the hardware. The ExecuteDomeCommand method in DomeDriver orchestrates this process, culminating in a call to StartActionWatchdog.
Sources: Driver/Pulsar_DomeDriver.cs:1584-1651
Starting the Watchdog
The StartActionWatchdog method in DomeDriver is responsible for instantiating and running the watchdog for a specific command.
// File: Driver/Pulsar_DomeDriver.cs:1653-1699
private void StartActionWatchdog(
TimeSpan timeout,
string action,
string longAction,
Func<ActionWatchdog.WatchdogResult> checkStatus,
string mqttTopic,
string mqttSuccess,
string mqttFail)
{
var cts = new CancellationTokenSource();
// ... dispose previous CTS ...
_actionWatchdogCts = cts;
var watchdog = new ActionWatchdog(
config: _config,
timeout: timeout,
action: action,
longAction: longAction,
checkStatus: checkStatus,
internalCts: cts,
resetRoutine: async () => await ResetRoutineAsync(), // Key part: links to recovery
mqttPublisher: _mqttPublisher,
mqttTopic: mqttTopic,
mqttSuccess: mqttSuccess,
mqttFail: mqttFail
);
_actionWatchdog = watchdog;
Task.Run(async () =>
{
try
{
await watchdog.Start();
}
// ... error handling and cleanup ...
});
}
This method configures the watchdog with a timeout, status-checking logic, and a crucial link to the ResetRoutineAsync method, which serves as the recovery mechanism.
Sources: Driver/Pulsar_DomeDriver.cs:1653-1699
Monitoring Loop
Once started, the ActionWatchdog.Start() method enters a loop to monitor the action. The following diagram illustrates this process.
This flowchart shows the main logic within the ActionWatchdog.Start() method.
Sources: Helper/ActionWatchdog.cs:60-141
Interaction with Dome Polling
The ActionWatchdog does not directly communicate with the hardware. Instead, it relies on the driver's main polling loop (PollLoopAsync) to update the dome's status in the shared _config object. The checkStatus delegate, provided by the DomeDriver, reads these updated values to determine if the action has completed.
The following sequence diagram shows the interaction for an OpenShutter command.
This diagram illustrates how the ActionWatchdog works in concert with the PollingThread to monitor the state of the DomeHardware.
Sources: Driver/Pulsar_DomeDriver.cs:511-534, 1701-1726, Helper/ActionWatchdog.cs:72-76
Failure Handling and Recovery
A key feature of the ActionWatchdog is its ability to trigger a recovery process upon failure or timeout.
When the watchdog's monitoring loop terminates with a Timeout, Failure, or Error result, it checks if a _resetRoutine was provided during its construction. If so, it invokes this routine.
In this driver, the resetRoutine is always a call to DomeDriver.ResetRoutineAsync(). This powerful mechanism allows the driver to attempt an automated recovery, which can involve:
- A "soft" reset by sending a
RESTARTcommand. - A "hard" reset by executing an external script/program to power-cycle the controller.
- Re-attempting the last command that failed.
This makes the driver resilient to hardware stalls and communication issues.
Sources: Helper/ActionWatchdog.cs:100-116, Driver/Pulsar_DomeDriver.cs:1078-1216, 1667