From 46bf1b140405ee53c4dc6070f63199d55e4054c9 Mon Sep 17 00:00:00 2001
From: Sora39831 <540587985@qq.com>
Date: Fri, 10 Apr 2026 10:33:53 +0800
Subject: [PATCH] docs: add multi-node shared control design spec

---
 ...-04-10-multi-node-shared-control-design.md | 357 ++++++++++++++++++
 1 file changed, 357 insertions(+)
 create mode 100644 docs/superpowers/specs/2026-04-10-multi-node-shared-control-design.md

diff --git a/docs/superpowers/specs/2026-04-10-multi-node-shared-control-design.md b/docs/superpowers/specs/2026-04-10-multi-node-shared-control-design.md
new file mode 100644
index 00000000..3f658d0c
--- /dev/null
+++ b/docs/superpowers/specs/2026-04-10-multi-node-shared-control-design.md
@@ -0,0 +1,357 @@
+# Multi-Node Shared Control Design
+
+## Context
+
+`3x-ui` already supports MariaDB as a database backend, but the runtime model remains single-node in practice: the panel reads persisted data, generates a local Xray configuration, and starts a local Xray process. Xray itself does not read MariaDB directly.
+
+The target capability is a minimal multi-node control model where multiple VPS nodes share account definitions and aggregate traffic through MariaDB, while each node still runs its own local Xray instance.
+
+This design intentionally keeps the existing panel pattern intact. It adds synchronization and role boundaries around the current database and Xray services instead of redesigning the runtime into a distributed cluster.
+
+## Goals
+
+- Support multiple nodes sharing account definitions through MariaDB.
+- Keep `master` as the only writer for shared account definitions.
+- Keep `worker` nodes running local Xray instances built from synchronized shared data.
+- Aggregate traffic from all nodes without last-write-wins corruption.
+- Preserve survivability when MariaDB is temporarily unavailable by using local cache files.
+- Minimize changes to the existing `3x-ui` architecture and operator workflow.
+
+## Non-Goals
+
+- No direct MariaDB access from Xray.
+- No leader election or automatic role failover.
+- No active-active shared account writes across nodes.
+- No push-based real-time config propagation.
+- No synchronization of panel-global settings such as panel port, web domain, TLS files, Telegram bot settings, or panel users.
+- No first-pass frontend-only read-only redesign for worker nodes.
+
+## Accepted Scope
+
+The first phase only synchronizes the shared-account surface:
+
+- inbounds and inbound client definitions
+- passwords and credential-bearing settings
+- quota, expiry, enabled/disabled state
+- aggregate upload and download counters
+
+The first phase does not synchronize node-local operational settings or observability data beyond minimal sync status metadata.
+
+## Recommended Approach
+
+Use a single-writer control model:
+
+- `master` is the only node allowed to mutate shared account definitions.
+- `worker` nodes keep the existing panel pages, but shared-account write requests are rejected in the backend.
+- all nodes, including `master`, continue to run local Xray instances and carry traffic
+- workers poll MariaDB for a shared account-set version and rebuild their local Xray state when that version changes
+- all nodes accumulate local traffic deltas and periodically flush them back to MariaDB using atomic increments
+
+This is the lowest-risk option because it keeps the current `controller -> service -> database -> local Xray` model intact while adding explicit control-plane boundaries.
+
+## Architecture
+
+### Runtime Model
+
+The system remains node-local at runtime:
+
+- each node runs its own `3x-ui` process
+- each node generates its own local Xray configuration
+- each node starts and restarts its own local Xray process
+- MariaDB acts only as the shared control database
+
+There is no direct node-to-node RPC. Synchronization happens indirectly through MariaDB and local background loops.
+
+### Role Boundaries
+
+`master` responsibilities:
+
+- accept and apply shared-account writes
+- bump the shared account-set version after successful writes
+- rebuild local Xray state immediately after successful writes
+- flush local traffic deltas like any other node
+
+`worker` responsibilities:
+
+- reject shared-account write requests in the backend
+- load synchronized shared account snapshots
+- rebuild local Xray state from the synchronized snapshot
+- flush local traffic deltas
+
+### Shared vs Local Data
+
+Shared through MariaDB:
+
+- shared inbound definitions
+- shared client definitions
+- shared quota and expiry state
+- aggregate traffic totals
+- synchronization metadata
+
+Kept local to each node:
+
+- panel port and path
+- web domain and TLS files
+- Telegram bot settings
+- panel users and login state
+- local cache and pending traffic files
+- local logs and node-specific operational state
+
+## Configuration Model
+
+Store node-control settings in `/etc/x-ui/x-ui.json`:
+
+- `nodeRole`: `master` or `worker`, default `master`
+- `nodeId`: unique node identifier, required for `worker`
+- `syncInterval`: shared snapshot poll interval in seconds, default `30`
+- `trafficFlushInterval`: traffic delta flush interval in seconds, default `10`
+
+Validation rules:
+
+- `nodeRole` must be `master` or `worker`
+- `worker` requires non-empty `nodeId`
+- `worker` requires `dbType = mariadb`
+- `syncInterval` must be positive
+- `trafficFlushInterval` must be positive
+
+Compatibility rules:
+
+- legacy configs without these keys must continue to load with defaults
+- existing grouped JSON layout rules for other settings must remain compatible
+- switching roles is a config change plus restart, not a schema migration
+
+## Data Model
+
+The first phase keeps shared business data in the existing tables and adds minimal synchronization metadata.
+
+### Shared Metadata Tables
+
+`shared_state`
+
+- `key` primary key
+- `version` monotonic int64
+- `updated_at`
+
+Reserved key:
+
+- `shared_accounts_version`
+
+`node_state`
+
+- `node_id` primary key
+- `node_role`
+- `last_sync_at`
+- `last_heartbeat_at`
+- `last_seen_version`
+- `last_error`
+- `updated_at`
+
+### Local Persistent Files
+
+`/etc/x-ui/shared-cache.json`
+
+- stores the last valid shared account snapshot
+- primarily used by workers for startup and MariaDB outage survivability
+- is not the system of record
+
+`/etc/x-ui/traffic-pending.json`
+
+- stores traffic deltas that have not yet been flushed successfully
+- is used by all nodes
+- prevents delta loss across retries or restarts
+
+### Versioning Strategy
+
+The design uses a single account-set version instead of per-record versions.
+
+Rule:
+
+- whenever `master` successfully changes shared account definitions, it increments `shared_accounts_version` in the same transaction
+
+This version answers only one question:
+
+- has the shared account set changed since the node last synchronized
+
+That keeps worker polling cheap and avoids per-row merge logic in the first phase.
+
+## Synchronization and Write Flow
+
+### Shared Account Write Flow
+
+For shared-account writes such as add, update, delete, enable, disable, quota change, expiry change, or client mutation:
+
+1. request enters the existing controller/service path
+2. service layer enforces `RequireMaster()`
+3. `worker` requests are rejected before any database or Xray mutation
+4. `master` applies the write
+5. the same transaction increments `shared_accounts_version`
+6. after transaction commit, `master` rebuilds local Xray configuration immediately
+
+The `master` node does not wait for polling to apply its own writes.
+
+### Worker Snapshot Sync Flow
+
+Workers synchronize on startup and on a fixed interval:
+
+1. load `shared-cache.json` if present
+2. perform an immediate version check against MariaDB
+3. if the shared version is newer, fetch the full shared account snapshot
+4. persist the snapshot to `shared-cache.json`
+5. rebuild local Xray configuration from that snapshot
+6. update `node_state` with sync status
+
+During steady state:
+
+- poll `shared_accounts_version` every `syncInterval`
+- if unchanged, only update node heartbeat/sync status
+- if changed, fetch the full snapshot and rebuild local Xray state
+
+### Traffic Flush Flow
+
+All nodes, including `master`, follow the same traffic accounting pattern:
+
+1. accumulate local per-account upload and download deltas
+2. persist pending deltas locally
+3. flush pending deltas every `trafficFlushInterval`
+4. apply deltas in MariaDB using atomic increments
+5. remove flushed deltas from local pending state after success
+
+Required rule:
+
+- nodes must never write absolute aggregate totals back to MariaDB
+
+Forbidden behavior:
+
+- reading the current total, adding locally, then overwriting the row
+- allowing concurrent nodes to race with last-write-wins semantics
+
+## Trigger Timing
+
+Synchronization is triggered by control-state changes and fixed intervals, not by direct node-to-node notifications.
+
+Triggers:
+
+- `master` shared-account write success: bump version in-transaction and rebuild local Xray immediately
+- worker startup: load local cache, then perform an immediate version check
+- worker periodic sync: poll version every `syncInterval`
+- all nodes periodic traffic flush: flush deltas every `trafficFlushInterval`
+- optional best-effort shutdown flush: attempt one final delta flush on graceful shutdown
+
+There is no direct push from `master` to `worker` in the first phase.
+
+## Failure Handling
+
+If MariaDB is temporarily unreachable:
+
+- workers continue using the last valid shared snapshot
+- traffic deltas remain in local pending storage
+- shared-account writes cannot proceed
+
+If `shared-cache.json` is missing or invalid and MariaDB is unreachable at worker startup:
+
+- do not construct a speculative snapshot
+- preserve the last successfully loaded runtime state if one already exists
+- record the error in node status
+
+If snapshot fetch succeeds but local Xray rebuild fails:
+
+- keep the last known-good local runtime configuration active
+- record the failure in `node_state.last_error`
+- retry on the next synchronization cycle
+
+If `master` writes shared state successfully but its own local rebuild fails:
+
+- MariaDB remains the source of truth with the new version
+- other nodes still synchronize from that committed state
+- `master` records the local rebuild error for operator action
+
+If a traffic flush fails:
+
+- keep the delta in pending storage
+- retry later
+- never drop pending deltas on flush failure
+
+## Testing Strategy
+
+### Config Tests
+
+Cover:
+
+- legacy config defaults
+- invalid `nodeRole`
+- `worker` without `nodeId`
+- `worker` with `sqlite`
+- non-positive interval validation
+
+### Service Tests
+
+Cover:
+
+- `RequireMaster()` enforcement
+- version bumping on successful shared-account writes
+- worker no-op behavior when version is unchanged
+- worker snapshot fetch and rebuild trigger when version changes
+- traffic flush success and retry behavior
+
+### Database Tests
+
+Cover:
+
+- migration of `shared_state` and `node_state`
+- seeding of `shared_accounts_version`
+- atomic increment semantics for traffic totals
+
+### Verification Expectations
+
+Phase-one acceptance:
+
+1. `master` writes are visible to workers within one poll cycle
+2. worker-side shared-account writes are rejected
+3. concurrent node traffic flushes do not lose aggregate totals
+4. workers continue operating from the last valid cache during temporary MariaDB outages
+
+## Operational Visibility
+
+Expose minimal operator-visible node information:
+
+- `nodeRole`
+- `nodeId`
+- `syncInterval`
+- `trafficFlushInterval`
+
+Record minimal per-node status in `node_state`:
+
+- last sync time
+- last heartbeat time
+- last seen version
+- last error
+
+This is sufficient for the first phase to answer:
+
+- whether a worker is lagging behind the current shared version
+- whether a node is alive but failing to synchronize
+- whether traffic flush retries are accumulating due to database problems
+
+## Implementation Boundary
+
+Expected change areas:
+
+- config loading and validation
+- startup validation and CLI setters
+- database model registration and metadata helpers
+- service-layer master-only guards
+- worker sync and cache services
+- traffic pending and flush services
+- startup wiring in the web server
+- operator-facing shell scripts and docs
+
+Areas intentionally unchanged in the first phase:
+
+- Xray protocol implementation
+- local Xray process ownership model
+- panel-global settings model
+- distributed leadership or push-based synchronization
+
+## Summary
+
+This design adds a minimal shared control plane to `3x-ui` without turning the system into a distributed cluster. `master` owns shared account writes, all nodes keep local Xray ownership, workers rebuild from synchronized snapshots, and traffic returns to MariaDB only as atomic deltas. The result is a bounded first phase that matches the existing architecture and keeps future extension paths open.