Architecture
Architecture
Deep-dive into the AxiomDB system design — the gateway, the data plane, tenant isolation, the Redis job queue, the circuit breaker, and the VPS topology.
System overview
┌────────────────────────────────────────────────────────────────┐
│ Clients │
│ Browser (dbs console) · axm CLI · REST API consumers │
└──────────────────────────┬─────────────────────────────────────┘
│ HTTPS / Bearer PASETO v4
┌──────────────────────────▼─────────────────────────────────────┐
│ AxiomDB Gateway (Rust 1.80 / Axum 0.7) │
│ │
│ Middleware stack: │
│ tracing ↔ compression ↔ CORS ↔ auth extractor │
│ │
│ Route groups: │
│ /api/v1/auth/* auth.rs │
│ /api/v1/network/* network.rs │
│ /api/v1/projects/* projects.rs │
│ /api/v1/jobs/* jobs.rs (provisioning_jobs) │
│ /api/v1/health main.rs │
│ │
│ Shared state (Arc<AppState>): │
│ db_pool → sqlx PgPool (control-plane) │
│ admin_pool → sqlx PgPool (BYPASSRLS role) │
│ redis → deadpool-redis connection pool │
│ config → Config struct (from env) │
│ idp_key → Ed25519 public key (cached from IdP) │
│ circuit_breaker → AtomicU32-based state machine │
└──────┬────────────────────────────────┬───────────────────────┘
│ sqlx async │ Redis XADD / XREADGROUP
┌──────▼──────────────────┐ ┌─────────▼─────────────────────────┐
│ Control-plane PostgreSQL │ │ Redis Stream │
│ (tenant-isolated, RLS) │ │ axiomdb:project_commands │
│ │ │ Consumer group: gateway_workers │
│ Tables: │ └─────────────┬─────────────────────┘
│ users │ │ XREAD
│ organizations │ ┌─────────────▼──────────────────┐
│ organization_members │ │ Tokio background task │
│ projects │ │ (started at boot in main.rs) │
│ project_databases │ │ Processes: provision, deprov, │
│ project_branches │ │ branch_create, branch_delete │
│ network_policies │ └─────────────┬──────────────────┘
│ network_rules │ │ exec
│ network_apply_events │ ┌─────────────▼──────────────────┐
│ provisioning_jobs │ │ square-dbctl (VPS binary) │
│ audit_events │ │ /usr/local/bin/square-dbctl │
│ credential_events │ │ │
│ cors_origins │ │ Commands: │
│ api_tokens │ │ provision --app X --env Y │
└───────────────────────────┘ │ branch-create --source ... │
│ allow-cidr --cidr ... │
│ stats --app X --env Y │
│ project-usage ... │
└─────────────┬──────────────────┘
│ SQL / system calls
┌─────────────▼──────────────────┐
│ Data-plane (VPS) │
│ PostgreSQL 14 │
│ PgBouncer (port 6432) │
│ pgBackRest │
│ fail2ban + crowdsec │
└────────────────────────────────┘Gateway internals
AppState
The gateway passes a single Arc<AppState> to all handlers:
pub struct AppState {
pub db_pool: PgPool, // control-plane queries
pub admin_pool: PgPool, // background worker pool (BYPASSRLS)
pub redis: deadpool_redis::Pool,
pub config: Config,
pub idp_key: Arc<RwLock<Option<String>>>, // cached Ed25519 key
pub circuit_breaker: Arc<CircuitBreaker>,
}Middleware chain
- TraceLayer — Request/response logging via
tracing - CompressionLayer — Gzip/Brotli response compression
- CORS — Origin allowlist read from the
cors_originscontrol-plane table at startup (refreshed on change) - Auth extractor — Extracts
ClaimsfromAuthorization: Bearerheader, checks both PASETO v4 (primary) and JWT HS256 (legacy fallback)
Auth extraction order
- Extract token from
Authorization: Bearerheader - Check if token starts with
v4.public.→ PASETO verification path - Fetch or use cached IdP public key (key ID from PASETO footer, fetched from
BASE_IDP_KEY_ENDPOINT) - Verify Ed25519 signature via PAE (Pre-Authentication Encoding)
- Validate claims:
iss,exp,nbf,token_use = "access" - If PASETO fails → try JWT HS256 fallback with
JWT_SECRET - Materialize the
Claimsstruct (sub, email, role, scopes, exp, tenant_id)
Provisioning pipeline
Client Gateway Redis Worker square-dbctl
│ │ │ │ │
│ POST /projects │ │ │ │
│──────────────────────▶│ │ │ │
│ │ validate input │ │ │
│ │ insert job row │ │ │
│ │ XADD stream │ │ │
│ │──────────────────▶│ │ │
│ 202 { job_id } │ │ │ │
│◀──────────────────────│ │ XREADGROUP │ │
│ │ │─────────────────▶│ │
│ │ │ │ exec provision │
│ │ │ │─────────────────▶│
│ │ │ │ │ CREATE DB
│ │ │ │ │ CREATE ROLE
│ │ │ │ stdout/exit code │
│ │ │ │◀─────────────────│
│ │ UPDATE jobs │ │ │
│ │◀────────────────────────────────────│ │
│ GET /jobs/:id (poll) │ │ │ │
│──────────────────────▶│ │ │ │
│ { status: succeeded } │ │ │ │
│◀──────────────────────│ │ │ │Tenant isolation implementation
Every handler that reads or writes a tenanted resource:
- Sets
app.current_tenantin the PostgreSQL session viaSET LOCAL:SELECT set_config('app.current_tenant', $1, true) - Includes
AND tenant_id = $Nin every queryWHEREclause. - PostgreSQL RLS acts as a second safety net via policies that check
current_setting('app.current_tenant', true).
The admin_pool connects as axiomdb_bg_worker (BYPASSRLS) and is only used for:
- Background provisioning jobs
- Schema migrations
- Tenant ID repair/update operations
Circuit breaker
All square-dbctl calls are wrapped in an async circuit breaker:
States:
Closed → All calls pass through
Open → Calls fail immediately with CircuitBreakerError::Open
HalfOpen → One probe call allowed; success → Closed, failure → Open
Configuration:
threshold: 5 consecutive failures → Open
timeout_secs: 60 seconds before transitioning Open → HalfOpen
Implementation:
AtomicU32 for state (no mutex, lock-free)
AtomicU32 for failure count
AtomicU64 for last_failure_time (UNIX seconds)When the gateway returns 503 Service Unavailable for a monitoring or provisioning endpoint, it means the circuit breaker is open. Wait 60 seconds and the probe will auto-run.
Control-plane database schema
See CONTROL_PLANE_SCHEMA.sql for the full base schema and the migrations directory for incremental changes.
Key schema additions (migration 004)
project_branches:parent_branch_id,is_default,protected,lifespan,expires_at,ttl_secondsorganizations+organization_members+organization_invitationsnetwork_policies+network_rules+network_apply_eventscredential_events
Migration 007 — Tenant isolation
- Added
tenant_id TEXT NOT NULLto all 9 tables - Enabled RLS on all tables
- Created
POLICY tenant_isolation_*for each table - Created
axiomdb_bg_workerrole withBYPASSRLS - Created composite indexes:
(tenant_id, status),(tenant_id, created_by, status)
VPS topology
VPS: 89.117.60.62
OS: Ubuntu 22.04 LTS
Users:
apps — runs Node.js / Next.js services (dbs console, docs)
opsdc — provisioning operations (NOSUPERUSER, NOCREATEDB)
Services (PM2):
axiomdb-ops — dbs Next.js ops console (port 3000)
axiomdb-docs — docs Next.js app (port 3020)
System services:
axiomdb-gateway — Rust binary (systemd, port 4060)
postgresql — PostgreSQL 14 (port 5432)
pgbouncer — Connection pool (port 6432)
redis — Job queue (port 6379, localhost only)
nginx — Reverse proxy (ports 80/443)
axiomdb.squareexp.com → localhost:3000
api.axiomdb.squareexp.com → localhost:4060
docs.axiomdb.squareexp.com → localhost:3020
fail2ban
crowdsec
Tools:
/usr/local/bin/square-dbctl — VPS provisioning binary
/usr/local/bin/pgbackrest — Backup tool
/home/opsdc/.creds/zone.env — Secret key storeSecurity design decisions
| Decision | Rationale |
|---|---|
| PASETO v4 over JWT RS256 | Shorter token, offline verification, implicit assertion prevents cross-system replay |
Tenant ID from IdP ctx claim | Centralized, unchangeable source of truth; prevents tenants from forging their own ID |
| Two-layer tenant isolation (app + RLS) | App-level is fast; RLS is a safety net if a query accidentally omits WHERE tenant_id |
BYPASSRLS only for admin pool | Minimal privilege; background workers need cross-tenant reads for provisioning |
Circuit breaker on square-dbctl | VPS instability should not cascade into API timeouts; fail-fast is better than slow-fail |
| Credential audit trail | Compliance and incident forensics — knowing who accessed credentials and when is critical |
How is this guide?