Common Issues

Relevant source files

The following files were used as context for generating this wiki page:

• Driver/Pulsar_DomeDriver.cs
• Helper/ActionWatchdog.cs
• Config/ConfigManager.cs
• Helper/SerialPortGuard.cs
• GNS/GNS.cs
• MQTT/MqttPublisher.cs
• UI/SettingsForm.cs
• Diagnostics/FileLogger.cs

The Pulsar Dome Driver is designed with a multi-layered system for detecting, handling, and recovering from common issues such as communication failures, command timeouts, and hardware stalls. This document outlines the architecture of these systems, from low-level serial communication guards to high-level automated reset routines.

The primary goal is to ensure robust, unattended operation by automatically recovering from transient errors. When an unrecoverable error occurs, the driver provides notifications through the Goodnight System (GNS) and MQTT. Understanding these mechanisms is key to diagnosing problems and configuring the driver for optimal stability.

Error Handling and Recovery Architecture

The driver employs a hierarchical approach to error handling. Issues are first handled at the lowest possible level (e.g., retrying a serial command). If an issue persists, it escalates to higher-level recovery mechanisms, culminating in a potential hardware reset.

This diagram illustrates the overall flow from a command request to potential recovery actions.

Level 1: Communication and Command Execution

At the lowest level, all communication with the dome controller is managed by the SerialPortGuard and the SendAndVerify method within the main driver.

SerialPortGuard

The SerialPortGuard class ensures that only one command is sent over the serial port at a time by using a SemaphoreSlim as a gate. It is responsible for writing the command and reading the response.

• It flushes the serial input buffer before sending a new command to clear any stale data.
• It implements a 750ms read timeout. If no response is received within this window, it throws an IOException, which is a primary trigger for error handling.

Sources: Helper/SerialPortGuard.cs:29-92

SendAndVerify

The DomeDriver.SendAndVerify method wraps the SerialPortGuard.Send call, adding a layer of response validation and retries.

• Response Modes: It can operate in several modes (Blind, Raw, MatchExact, MatchAny) to validate the controller's response.
• Retries: It will retry a command up to 2 times by default if the initial response is not what was expected.
• Failure: If the command fails after all retries, the IsMatch property of the returned ResponseResult will be false. In critical command paths like ExecuteDomeCommand, this failure triggers the RaiseAlarmAndReset function, escalating the error.

*Sources: Driver/Pulsar_DomeDriver.cs:1750-1836, Helper/ResponseResult.cs

Level 2: Action Monitoring with ActionWatchdog

For long-running actions like opening the shutter or slewing the dome, a simple command-response check is insufficient. The ActionWatchdog class monitors the state of the dome over time to ensure the action completes successfully.

When a command like OpenShutter is issued, an ActionWatchdog instance is created and started. It continuously polls a checkStatus function to determine the outcome.

The watchdog has several possible results:

Result	Description	Consequence
InProgress	The action is still executing within the timeout period.	The watchdog continues to monitor.
Success	The checkStatus function returned a success condition (e.g., _config.ShutterStatus == 0 for OpenShutter).	The watchdog stops, and the driver returns to an idle state.
Timeout	The action did not complete within the configured timeout (e.g., ShutterTimeout).	The watchdog invokes the ResetRoutineAsync.
Failure / Error	The checkStatus function returned an explicit failure or error state (e.g., _config.ShutterStatus == 4).	The watchdog invokes the ResetRoutineAsync.

Sources: Helper/ActionWatchdog.cs:30-35, 100-120

The following diagram shows the state transitions for the ActionWatchdog.

Level 3: System Health and Polling Watchdog

To ensure the driver itself remains responsive, two background processes are active when connected: the polling loop and a system watchdog that monitors the poller.

Polling Loop (PollLoopAsync)

This loop runs continuously, calling DomeStatus() at a configurable interval (_pollingIntervalMs). DomeStatus() queries the controller for its complete status (azimuth, shutter state, etc.) and updates the driver's internal state variables in the ConfigManager.

• It is responsible for detecting when an action has completed successfully (e.g., when _config.DomeState becomes 0 after a slew).
• If polling encounters 3 consecutive errors, it throttles itself to avoid spamming a non-responsive controller.
• The polling loop is intentionally stopped during command execution to prevent interference.

Sources: Driver/Pulsar_DomeDriver.cs:644-741

System Watchdog (StartWatchdog)

A separate watchdog task monitors the _lastPollTimestamp, which is updated by the polling loop.

• It checks every 5 seconds (_systemWatchdogInterval).
• If the time since the last successful poll exceeds a threshold (10 seconds, _pollingStallThreshold), it assumes the polling loop has stalled.
• If a stall is detected, the watchdog forcibly stops and restarts the polling loop to recover.
• It also publishes a periodic MQTT heartbeat message to the Dome/Watchdog topic.

This diagram shows the logic of the system watchdog.

Level 4: The Automated Reset Routine

When the ActionWatchdog times out or a critical command fails, the ResetRoutineAsync is triggered. This is the driver's most powerful recovery mechanism. It can perform a soft (command-based) reset or a hard (power-cycle) reset.

Reset Logic Flow

1. Stop Polling: The routine immediately stops the polling loop to take exclusive control.
2. Attempt Soft Reset: If enabled (SoftReset is true in config), it sends a RESTART command to the controller. It then waits and monitors the shutter status to confirm a successful reboot.
3. Attempt Hard Reset: If the soft reset is disabled, not attempted, or fails, the routine proceeds to a hard reset (if enabled via HardReset config).
4. Hard Reset Process:
   • The driver disconnects from the serial port.
   • It launches an external executable (ResetExe) with "OFF" parameters (ResetOffParameters) to power down the controller.
   • It waits for a CycleDelay.
   • It launches the same executable with "ON" parameters (ResetOnParameters) to power the controller back on.
   • It waits for a ResetDelay for the controller to initialize.
   • It attempts to reconnect and ping the controller.
5. Replay Command: If either reset is successful, the driver attempts to re-issue the last command that failed (ReplayLastCommand).
6. Unrecoverable Failure: If the hard reset is attempted and fails, the driver logs an unrecoverable failure and sends a high-priority alarm.

Sources: Driver/Pulsar_DomeDriver.cs:1309-1453

Reset Configuration

The reset behavior is highly configurable via the settings dialog.

Setting	Description	File/Class
SoftReset	Enables/disables the software RESTART command.	ConfigManager, SettingsForm
HardReset	Enables/disables the external hardware reset process.	ConfigManager, SettingsForm
ResetExe	Path to the external executable used for power cycling.	ConfigManager, SettingsForm
ResetOffParameters	Command-line arguments for the ResetExe to turn power OFF.	ConfigManager, SettingsForm
ResetOnParameters	Command-line arguments for the ResetExe to turn power ON.	ConfigManager, SettingsForm
ResetDelay	Time (ms) to wait for the controller to boot after a hard reset.	ConfigManager, SettingsForm
CycleDelay	Time (ms) to wait between power-off and power-on commands.	ConfigManager, SettingsForm

Sources: Config/ConfigManager.cs:113-189, UI/SettingsForm.cs:72-97

Error Notification

When a significant error occurs, especially one that triggers a reset, the driver notifies external systems.

• GNS (Goodnight System): The GNS.SendGNS method is called to send messages over a named pipe (GNSBridge). It can send different types of messages, including Alarm, Message, and Stop. This is controlled by the UseGNS setting.
• MQTT: The MqttPublisher class is used to publish messages to an MQTT broker. Alarms are typically sent to the Dome/Alarm topic, while status updates go to other topics. This is controlled by the UseMQTT setting.

The central method RaiseAlarmAndReset is called in response to a critical failure. It is responsible for sending notifications to both GNS and MQTT before initiating the ResetRoutineAsync.

Sources: Driver/Pulsar_DomeDriver.cs:2176-2184, GNS/GNS.cs:26-47, MQTT/MqttPublisher.cs:91-115