Centralized Backup Orchestration and Rclone-based Gateway

[DavidePrincipi]

This proposal revises the current NS8 backup architecture by moving scheduling and orchestration from individual applications to the node level, with the goal of reducing resource usage, simplifying destination support, and improving consistency.

Current architecture
Each NS8 application runs Restic independently and pushes backups directly to a configured Destination (e.g. S3 bucket), according to one or more Schedules implemented as app-specific systemd timer units.

Current limitations

• All application timers typically trigger at the same time on a node, causing multiple backups to run in parallel and leading to high memory and CPU consumption.
• Adding support for additional cloud backup providers requires code changes. At present, Restic is used directly for AWS, Azure, and B2, while SMB and generic S3 destinations rely on Restic behind rclone serve, resulting in duplicated logic and inconsistent handling.

Proposed architecture

• Backup Schedules are implemented as systemd timers at the node-agent level. When a schedule triggers, the node agent runs application backups sequentially on that node, impersonating each app as needed (e.g. via runagent -m appX) to keep resource usage under control.
• All backup destinations are accessed through Rclone, with support for importing a raw rclone.conf file to allow flexible and extensible backend configuration without code changes.
• Applications send backup data through a node-level gateway implemented with rclone serve rest. Applications authenticate to the gateway and are granted append-only Restic access to their backup destinations.
• After backup execution, the node agent is responsible for running Restic prune operations, enforcing the configured retention policy and cleaning up old snapshots centrally.

This change centralizes backup scheduling and retention logic, reduces peak resource usage on nodes, and standardizes destination handling across all applications.