[FEATURE] Support high availability and multi-instance server deployments #133

Description

The Voltius server currently supports only a single active instance per database.

This is an intentional architectural restriction because several important components maintain process-local state, including realtime notifications, presence, terminal sessions, and rate limiting.

Supporting multiple coordinated server instances would improve availability, enable rolling deployments, and provide a foundation for horizontal scaling.

This is a long-term architectural enhancement rather than a bug report.

Current behavior

The server uses a PostgreSQL advisory lock in src/single_instance.rs to prevent multiple instances from operating against the same database.

A second server instance normally refuses to start.

The ALLOW_MULTIPLE_INSTANCES=true setting bypasses this restriction but does not make multi-instance operation safe.

Relevant limitations include:

• Realtime notifications: SyncNotifier uses an in-process Tokio broadcast channel.
• Presence: Online status and connection usage rely on process-local maps.
• Shared terminal sessions: TerminalManager maintains live session state and communication channels in memory.
• Rate limiting: Certain limits are enforced through process-local state.

Consequently, clients connected to different server instances may observe inconsistent state or miss notifications.

Simply placing multiple servers behind a load balancer would not resolve these limitations.

Proposed feature

Introduce support for coordinated multi-instance deployments while preserving the existing single-instance configuration as the simple, supported default.

Potential capabilities:

• Multiple server instances using a shared PostgreSQL database.
• Cross-instance distribution of realtime events.
• Consistent presence and connection-usage state.
• Coordinated rate limiting.
• Defined ownership and routing of live terminal sessions.
• Health checks suitable for load balancers.
• Graceful shutdown and connection draining.
• Safe rolling deployments.

The implementation should not require additional infrastructure for users running a single server.

Suggested architecture

1. Distributed event delivery

Replace or extend the process-local notification mechanism with a cross-instance event transport.

Potential options include:

• PostgreSQL LISTEN/NOTIFY for lightweight signaling.
• Redis Pub/Sub or Redis Streams.
• Another suitable event bus.

The selected mechanism should consider event ordering, recipient routing, delivery guarantees, reconnect behavior, and recovery after missed notifications.

SSE clients should receive relevant notifications regardless of which instance handled the originating request.

2. Distributed presence and connection usage

Move the authoritative presence state away from individual process-local maps.

Presence should account for:

• Multiple simultaneously connected devices.
• Multiple SSE connections per user.
• Connections distributed across server instances.
• Abrupt client and server disconnects.
• Expiration of stale presence records.

A lease-based mechanism with expiration and refresh could prevent indefinitely stale entries after crashes.

Connection-usage state should follow equivalent consistency rules.

3. Shared terminal sessions

Shared terminal sessions require special consideration because their live communication channels are currently owned by one server process.

Possible approaches include:

• Assigning each session to an owning instance.
• Routing participants to the owning instance.
• Using an inter-instance relay for session traffic.
• Maintaining a shared session directory and lifecycle state.

The design should explicitly define what happens when the owning instance crashes.

Transparent survival of an active terminal session across a server crash should not be assumed unless the necessary state and transport are actually recoverable.

An initial implementation could terminate affected sessions cleanly and allow clients to reconnect or recreate them.

4. Coordinated rate limiting

Rate limits should be enforced consistently regardless of which instance receives a request.

A shared backend may be necessary for limits that are currently maintained in process memory.

Authorization and licensing checks should retain their existing semantics.

5. Load balancing and graceful shutdown

Document the requirements for reverse proxies and load balancers.

Support should include:

• Long-lived SSE connections.
• WebSocket connections used for terminal collaboration.
• Health and readiness checks.
• Graceful shutdown.
• Draining of active connections during deployment.
• Reconnection behavior when an instance becomes unavailable.

Session affinity may help with some connection types but should not replace proper state coordination.

Deployment modes

Consider supporting two documented modes.

Single instance (default)

• Current architecture remains supported.
• No Redis or additional services required.
• Existing deployment files continue working.

Multi-instance / HA (optional)

• Explicitly enabled.
• Shared coordination services configured.
• Multiple application instances supported.
• Load balancer placed in front of the instances.
• Documented failover and recovery behavior.

The existing single-instance safety guard should remain enabled until the distributed components are ready.

Incremental implementation

This feature could be delivered in stages:

1. Define the consistency and failover requirements.
2. Introduce an abstraction for cross-instance notifications.
3. Implement distributed presence and connection-usage tracking.
4. Coordinate rate limiting.
5. Introduce session ownership and routing.
6. Add readiness and graceful shutdown behavior.
7. Validate operation behind a load balancer.
8. Enable supported multi-instance deployments.

Each stage should remain testable without requiring the entire HA architecture to be completed immediately.

Regression and integration tests

Please consider covering:

• Two server instances connected to the same database.
• A sync event originating on instance A reaching an SSE client on instance B.
• Team membership changes propagated across instances.
• Multiple devices connected through different instances.
• Presence cleanup after an instance crashes.
• Connection-usage updates across instances.
• Consistent rate limiting across instances.
• Shared terminal session access through different instances.
• Session owner failure.
• Rolling restart of one instance.
• Network partition between coordination components.
• Recovery after the event transport becomes temporarily unavailable.
• Existing single-instance deployment behavior.

Security considerations

Multi-instance operation must preserve existing authorization and end-to-end encryption guarantees.

Shared infrastructure must not gain access to plaintext vault secrets or terminal content.

Terminal routing, presence information, and session ownership metadata should be scoped and authenticated appropriately.

Cross-instance communication must not allow one client to access another user's session or team data.

Benefits

• Improved service availability.
• Reduced downtime during deployments.
• Foundation for horizontal scaling.
• Better resilience against individual application-instance failures.
• More deployment flexibility for self-hosted and enterprise environments.

Scope and priority

This is a long-term architectural proposal.

It does not request removing the current single-instance restriction before the underlying state coordination is implemented.

It also does not require full active-active clustering as the first milestone.

A documented active/passive failover design could be considered as an intermediate step.

Verification

Based on source-level review of the server main branch.

The server explicitly prevents multiple active instances because realtime notifications, terminal sessions, rate limits, and presence currently depend on process-local state.

This is a feature request for a currently unsupported deployment architecture, not a confirmed runtime defect.