3x-ui/docs/superpowers/specs/2026-04-10-multi-node-shared-control-design.md

# Multi-Node Shared Control Design

## Context

`3x-ui` already supports MariaDB as a database backend, but the runtime model remains single-node in practice: the panel reads persisted data, generates a local Xray configuration, and starts a local Xray process. Xray itself does not read MariaDB directly.

The target capability is a minimal multi-node control model where multiple VPS nodes share account definitions and aggregate traffic through MariaDB, while each node still runs its own local Xray instance.

This design intentionally keeps the existing panel pattern intact. It adds synchronization and role boundaries around the current database and Xray services instead of redesigning the runtime into a distributed cluster.

## Goals

- Support multiple nodes sharing account definitions through MariaDB.
- Keep `master` as the only writer for shared account definitions.
- Keep `worker` nodes running local Xray instances built from synchronized shared data.
- Aggregate traffic from all nodes without last-write-wins corruption.
- Preserve survivability when MariaDB is temporarily unavailable by using local cache files.
- Minimize changes to the existing `3x-ui` architecture and operator workflow.

## Non-Goals

- No direct MariaDB access from Xray.
- No leader election or automatic role failover.
- No active-active shared account writes across nodes.
- No push-based real-time config propagation.
- No synchronization of panel-global settings such as panel port, web domain, TLS files, Telegram bot settings, or panel users.
- No first-pass frontend-only read-only redesign for worker nodes.

## Accepted Scope

The first phase only synchronizes the shared-account surface:

- inbounds and inbound client definitions
- passwords and credential-bearing settings
- quota, expiry, enabled/disabled state
- aggregate upload and download counters

The first phase does not synchronize node-local operational settings or observability data beyond minimal sync status metadata.

## Recommended Approach

Use a single-writer control model:

- `master` is the only node allowed to mutate shared account definitions.
- `worker` nodes keep the existing panel pages, but shared-account write requests are rejected in the backend.
- all nodes, including `master`, continue to run local Xray instances and carry traffic
- workers poll MariaDB for a shared account-set version and rebuild their local Xray state when that version changes
- all nodes accumulate local traffic deltas and periodically flush them back to MariaDB using atomic increments

This is the lowest-risk option because it keeps the current `controller -> service -> database -> local Xray` model intact while adding explicit control-plane boundaries.

## Architecture

### Runtime Model

The system remains node-local at runtime:

- each node runs its own `3x-ui` process
- each node generates its own local Xray configuration
- each node starts and restarts its own local Xray process
- MariaDB acts only as the shared control database

There is no direct node-to-node RPC. Synchronization happens indirectly through MariaDB and local background loops.

### Role Boundaries

`master` responsibilities:

- accept and apply shared-account writes
- bump the shared account-set version after successful writes
- rebuild local Xray state immediately after successful writes
- flush local traffic deltas like any other node

`worker` responsibilities:

- reject shared-account write requests in the backend
- load synchronized shared account snapshots
- rebuild local Xray state from the synchronized snapshot
- flush local traffic deltas

### Shared vs Local Data

Shared through MariaDB:

- shared inbound definitions
- shared client definitions
- shared quota and expiry state
- aggregate traffic totals
- synchronization metadata

Kept local to each node:

- panel port and path
- web domain and TLS files
- Telegram bot settings
- panel users and login state
- local cache and pending traffic files
- local logs and node-specific operational state

## Configuration Model

Store node-control settings in `/etc/x-ui/x-ui.json`:

- `nodeRole`: `master` or `worker`, default `master`
- `nodeId`: unique node identifier, required for `worker`
- `syncInterval`: shared snapshot poll interval in seconds, default `30`
- `trafficFlushInterval`: traffic delta flush interval in seconds, default `10`

Validation rules:

- `nodeRole` must be `master` or `worker`
- `worker` requires non-empty `nodeId`
- `worker` requires `dbType = mariadb`
- `syncInterval` must be positive
- `trafficFlushInterval` must be positive

Compatibility rules:

- legacy configs without these keys must continue to load with defaults
- existing grouped JSON layout rules for other settings must remain compatible
- switching roles is a config change plus restart, not a schema migration

## Data Model

The first phase keeps shared business data in the existing tables and adds minimal synchronization metadata.

### Shared Metadata Tables

`shared_state`

- `key` primary key
- `version` monotonic int64
- `updated_at`

Reserved key:

- `shared_accounts_version`

`node_state`

- `node_id` primary key
- `node_role`
- `last_sync_at`
- `last_heartbeat_at`
- `last_seen_version`
- `last_error`
- `updated_at`

### Local Persistent Files

`/etc/x-ui/shared-cache.json`

- stores the last valid shared account snapshot
- primarily used by workers for startup and MariaDB outage survivability
- is not the system of record

`/etc/x-ui/traffic-pending.json`

- stores traffic deltas that have not yet been flushed successfully
- is used by all nodes
- prevents delta loss across retries or restarts

### Versioning Strategy

The design uses a single account-set version instead of per-record versions.

Rule:

- whenever `master` successfully changes shared account definitions, it increments `shared_accounts_version` in the same transaction

This version answers only one question:

- has the shared account set changed since the node last synchronized

That keeps worker polling cheap and avoids per-row merge logic in the first phase.

## Synchronization and Write Flow

### Shared Account Write Flow

For shared-account writes such as add, update, delete, enable, disable, quota change, expiry change, or client mutation:

1. request enters the existing controller/service path
2. service layer enforces `RequireMaster()`
3. `worker` requests are rejected before any database or Xray mutation
4. `master` applies the write
5. the same transaction increments `shared_accounts_version`
6. after transaction commit, `master` rebuilds local Xray configuration immediately

The `master` node does not wait for polling to apply its own writes.

### Worker Snapshot Sync Flow

Workers synchronize on startup and on a fixed interval:

1. load `shared-cache.json` if present
2. perform an immediate version check against MariaDB
3. if the shared version is newer, fetch the full shared account snapshot
4. persist the snapshot to `shared-cache.json`
5. rebuild local Xray configuration from that snapshot
6. update `node_state` with sync status

During steady state:

- poll `shared_accounts_version` every `syncInterval`
- if unchanged, only update node heartbeat/sync status
- if changed, fetch the full snapshot and rebuild local Xray state

### Traffic Flush Flow

All nodes, including `master`, follow the same traffic accounting pattern:

1. accumulate local per-account upload and download deltas
2. persist pending deltas locally
3. flush pending deltas every `trafficFlushInterval`
4. apply deltas in MariaDB using atomic increments
5. remove flushed deltas from local pending state after success

Required rule:

- nodes must never write absolute aggregate totals back to MariaDB

Forbidden behavior:

- reading the current total, adding locally, then overwriting the row
- allowing concurrent nodes to race with last-write-wins semantics

## Trigger Timing

Synchronization is triggered by control-state changes and fixed intervals, not by direct node-to-node notifications.

Triggers:

- `master` shared-account write success: bump version in-transaction and rebuild local Xray immediately
- worker startup: load local cache, then perform an immediate version check
- worker periodic sync: poll version every `syncInterval`
- all nodes periodic traffic flush: flush deltas every `trafficFlushInterval`
- optional best-effort shutdown flush: attempt one final delta flush on graceful shutdown

There is no direct push from `master` to `worker` in the first phase.

## Failure Handling

If MariaDB is temporarily unreachable:

- workers continue using the last valid shared snapshot
- traffic deltas remain in local pending storage
- shared-account writes cannot proceed

If `shared-cache.json` is missing or invalid and MariaDB is unreachable at worker startup:

- do not construct a speculative snapshot
- preserve the last successfully loaded runtime state if one already exists
- record the error in node status

If snapshot fetch succeeds but local Xray rebuild fails:

- keep the last known-good local runtime configuration active
- record the failure in `node_state.last_error`
- retry on the next synchronization cycle

If `master` writes shared state successfully but its own local rebuild fails:

- MariaDB remains the source of truth with the new version
- other nodes still synchronize from that committed state
- `master` records the local rebuild error for operator action

If a traffic flush fails:

- keep the delta in pending storage
- retry later
- never drop pending deltas on flush failure

## Testing Strategy

### Config Tests

Cover:

- legacy config defaults
- invalid `nodeRole`
- `worker` without `nodeId`
- `worker` with `sqlite`
- non-positive interval validation

### Service Tests

Cover:

- `RequireMaster()` enforcement
- version bumping on successful shared-account writes
- worker no-op behavior when version is unchanged
- worker snapshot fetch and rebuild trigger when version changes
- traffic flush success and retry behavior

### Database Tests

Cover:

- migration of `shared_state` and `node_state`
- seeding of `shared_accounts_version`
- atomic increment semantics for traffic totals

### Verification Expectations

Phase-one acceptance:

1. `master` writes are visible to workers within one poll cycle
2. worker-side shared-account writes are rejected
3. concurrent node traffic flushes do not lose aggregate totals
4. workers continue operating from the last valid cache during temporary MariaDB outages

## Operational Visibility

Expose minimal operator-visible node information:

- `nodeRole`
- `nodeId`
- `syncInterval`
- `trafficFlushInterval`

Record minimal per-node status in `node_state`:

- last sync time
- last heartbeat time
- last seen version
- last error

This is sufficient for the first phase to answer:

- whether a worker is lagging behind the current shared version
- whether a node is alive but failing to synchronize
- whether traffic flush retries are accumulating due to database problems

## Implementation Boundary

Expected change areas:

- config loading and validation
- startup validation and CLI setters
- database model registration and metadata helpers
- service-layer master-only guards
- worker sync and cache services
- traffic pending and flush services
- startup wiring in the web server
- operator-facing shell scripts and docs

Areas intentionally unchanged in the first phase:

- Xray protocol implementation
- local Xray process ownership model
- panel-global settings model
- distributed leadership or push-based synchronization

## Summary

This design adds a minimal shared control plane to `3x-ui` without turning the system into a distributed cluster. `master` owns shared account writes, all nodes keep local Xray ownership, workers rebuild from synchronized snapshots, and traffic returns to MariaDB only as atomic deltas. The result is a bounded first phase that matches the existing architecture and keeps future extension paths open.