fix: MQTT reconnection and reliability hardening

[eman]

Second batch from the comprehensive repo evaluation: fixes the reliability cluster that could leave a long-running client permanently offline with all failures silently swallowed.

Bugs fixed

• Reconnection loop died on auth errors (mqtt/reconnection.py): the backoff loop caught only (AwsCrtError, RuntimeError), but the quick/deep reconnect paths raise TokenRefreshError, AuthenticationError, and MqttCredentialsError — none of which are RuntimeError. A network outage coinciding with token expiry killed the reconnect task with an unretrieved exception; the client never reconnected despite unlimited retries. All Nwp500Errors (and TimeoutError) now count as failed attempts and are retried. Only InvalidCredentialsError is fatal: it stops the loop and emits reconnection_failed.
• disconnect() was a no-op while interrupted (mqtt/client.py): calling disconnect() while _connected was False returned early without disabling reconnection or stopping periodic tasks — the backoff loop would resurrect the connection after shutdown (the CLI's finally: await mqtt.disconnect() hit exactly this path after a drop). Now reconnection is always disabled, periodic tasks stopped, and the SDK connection torn down via close() even when not connected.
• Queued commands lost after active/deep reconnect (mqtt/client.py): the queue was only flushed from the SDK's on_connection_resumed callback, which never fires for the brand-new connection built by active/deep reconnection. Both paths now flush after resubscribe_all().
• Periodic tasks silently killed by MQTT errors (mqtt/periodic.py): MqttNotConnectedError/MqttPublishError raised by a publish racing a disconnect escaped the loop's catch and permanently ended polling while the task still looked alive.
• Dropped run_coroutine_threadsafe futures (mqtt/client.py): exceptions from scheduled coroutines (e.g. failed resubscribe after clean-session resume, leaving the client connected but deaf) vanished into discarded futures. A done-callback now logs them.
• CancelledError swallowed with break (reconnection + periodic loops): cancelled tasks ended "successfully" instead of cancelled, and the reconnection loop could continue past the loop and emit reconnection_failed during a manual disconnect. Cancellation now propagates.

Improvements

• Operation ack timeouts (mqtt/connection.py, mqtt/subscriptions.py): connect/publish/subscribe/unsubscribe/disconnect acknowledgements are awaited with MqttConnectionConfig.operation_timeout (default 30s). Previously a half-open TCP connection could hang callers until the 20-minute keep-alive expired. The awscrt future stays shielded, so timeouts/cancellation never corrupt SDK callbacks.
• Backoff jitter: reconnect delays are randomized (±50%, capped at max_reconnect_delay) to prevent synchronized fleet-wide reconnect storms after AWS IoT's 24-hour disconnect.

Tests

New tests/test_mqtt_reliability.py (17 tests): auth/MQTT errors retried, InvalidCredentialsError fatal + event, cancellation marks tasks cancelled, disconnect-while-interrupted shuts everything down, queue flush after active and deep reconnect, periodic loop survives MqttError, scheduled-coroutine exception logging, publish/subscribe ack timeouts, jitter bounded by max_reconnect_delay.

Validation

• pytest: 525 passed (3 consecutive runs, no flakiness)
• ruff check / ruff format --check: clean
• pyright src/nwp500: 0 errors
• mypy src/nwp500: no issues

[Copilot]

Pull request overview

This PR hardens the MQTT client’s long-running reliability by making reconnection/periodic loops resilient to auth/MQTT exceptions, ensuring disconnect fully shuts down background activity, and adding bounded operation ACK timeouts to avoid indefinite hangs on half-open connections.

Changes:

• Make reconnection and periodic loops retry on broader library errors (and propagate CancelledError correctly), with bounded backoff jitter.
• Add broker acknowledgement timeouts for connect/publish/subscribe/unsubscribe/disconnect via MqttConnectionConfig.operation_timeout.
• Add a comprehensive regression test suite covering the above reliability scenarios and document changes in the changelog.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_mqtt_reliability.py	New regression tests covering reconnection, disconnect semantics, periodic loop resilience, scheduled coroutine logging, and ACK timeouts.
src/nwp500/mqtt/utils.py	Adds operation_timeout to MQTT connection configuration.
src/nwp500/mqtt/subscriptions.py	Applies subscribe/unsubscribe ACK timeouts and stores an operation_timeout setting.
src/nwp500/mqtt/reconnection.py	Broadens retryable exception set, adds jittered backoff, and fixes cancellation propagation/fatal credential handling.
src/nwp500/mqtt/periodic.py	Ensures CancelledError propagates and periodic polling survives broader MQTT/library errors.
src/nwp500/mqtt/connection.py	Introduces a shared _await_ack() helper to await SDK ACK futures with timeout/shielding.
src/nwp500/mqtt/client.py	Ensures queued commands flush after active/deep reconnect, hardens disconnect during interruption, and logs thread-scheduled coroutine failures.
CHANGELOG.rst	Documents the reliability fixes and the new operation timeout / jitter behavior.

Comments suppressed due to low confidence (1)

src/nwp500/mqtt/utils.py:250

• operation_timeout is a user-facing config value, but __post_init__ doesn’t validate it. asyncio.wait_for(..., timeout<=0) will time out immediately, which is surprising and hard to debug. Consider enforcing a positive value (or explicitly supporting None to disable timeouts).

    # MQTT operation acknowledgement timeout
    operation_timeout: float = 30.0  # seconds

    # Command queue settings
    enable_command_queue: bool = True
    max_queued_commands: int = 100

    def __post_init__(self) -> None:
        """Generate client ID if not provided and validate settings."""
        if not self.client_id:
            object.__setattr__(
                self, "client_id", f"navien-client-{uuid.uuid4().hex[:8]}"
            )
        if self.deep_reconnect_threshold < 1:
            object.__setattr__(self, "deep_reconnect_threshold", 1)