Tenancy, BYOK & operations

Parent: Architecture · Covers roadmap phase 8 plus cross-cutting concerns wired in from phase 0. Status: as built — phase 8 shipped in felix-app (multi-user enabled for real); as-built notes are inline.

1. Scope and goals

The shared-vs-per-user boundary made operational: the LLM gateway and model tiers, BYOK key handling, budgets and the usage ledger, data isolation (Postgres RLS + lake prefixes), secrets, observability, the eval harness as CI, backups, and deployment topology.

Goals: swap any model by config; every LLM dollar attributed to a user and purpose; a user's key never leaves the vault unencrypted except per-request; one user can never read another's rows, prefixes, or traces; single-user today runs on the exact seams multi-user will use.

Non-goals (v1): SOC2-style formality, per-prefix object-store IAM (single app credential v1), SSO.

2. Gateway and model tiers

Self-hosted LiteLLM proxy (own container + its own small Postgres schema — its virtual-key/budget tables can live in the same Supabase Postgres under a separate schema, one database to operate instead of two). The brain speaks OpenAI-compatible to the proxy only — providers are proxy config.

Model aliases (swap = config edit, zero code): embed (embeddings), cheap (story synthesis, summaries, generic market recap), mid (daily generation), best (chat, ticker takes). Each alias maps to a concrete provider model with fallbacks. As built (phase 4): the mapping is fully provider-agnostic — each tier's concrete model is env-driven ({TIER}_MODEL) and the model's provider/ prefix picks the key, so a provider swap is a .env edit with zero code change; the dev default block runs the whole gateway on a free Google AI Studio (Gemini) key, with Anthropic-class tiers the prod intent.

Keys:

House keys (env/secrets → proxy config): used by shared jobs — story synthesis, generic recap, embeddings.
Virtual keys: one per user via /key/generate with user_id metadata, max_budget + budget_duration: 30d, tpm/rpm caps. All per-user calls authenticate with the virtual key → spend tracked per user by the proxy. As built (phase 8): provisioning is lazy get-or-create on a user's first metered call (advisory-locked against concurrent first-calls minting duplicates), not a signup webhook; the proxy's own tables live in a litellm schema in the same Supabase Postgres.
BYOK: user pastes a provider key in Settings → secret material goes into Supabase Vault (authenticated encryption at rest, encryption key held in Supabase's backend outside the database so dumps/replicas stay encrypted; the decrypted view is granted to the brain's role only; display last4 only; never logged), with metadata in doc 03's api_keys row (provider, last4, status, Vault reference). Per request, the brain reads the key through the Vault view and attaches the provider key header; the proxy runs with forward_llm_provider_auth_headers: true so the user's key is forwarded to the provider while the virtual key still authenticates and meters the call. Alternative considered: registering per-user model deployments inside LiteLLM — rejected: keys would live in proxy state, and key rotation/removal becomes a proxy-config problem instead of a vault row.
BYOK provider routing: keys are stored per provider (doc 03 api_keys, unique per user × provider v1), so alias resolution must be per user — BYOK-funded calls resolve every tier alias within the user's key's provider (an Anthropic-only user gets Anthropic models at every tier), pinned via the virtual key's allowed-models/routing metadata. An alias resolving to a provider the user holds no key for would fail auth or silently bill the house key; cross-provider fallbacks are disabled for BYOK-funded calls for the same reason. Pre-launch checklist: verify per-provider key-precedence behavior on the routed /chat/completions path (LiteLLM's auth-header forwarding has provider-specific sharp edges), and audit the proxy's spend/request logs to confirm forwarded provider keys never land in request-tracking metadata.

Fallback policy: daily briefing falls back to the house key (small, capped per user) if BYOK is missing/invalid — the product never silently stops; the briefing notes it in the published text. Chat requires BYOK for non-owner users (friendly refusal pointing at Settings → Keys). As built (phase 8): this is no longer a future posture — multi-user is enabled for real (MULTI_USER=true on the running instance, a real member user as a standing fixture). The owner is house-funded; the house-key-v1 → per-user-virtual-keys cutover happened, and MULTI_USER=false remains wire-identical to the phase-7 behavior. Funding kind is stamped at call time (key_kind on every ai_runs row) rather than derived later from user_id-nullness — the derivation would under-count the house cap once owner-house-funding shipped.

3. Cost attribution

Two records per call: the proxy's spend log (by virtual key) and the brain's ai_runs row (purpose, model, tokens, cost estimate). A nightly reconciliation job cross-checks them into usage_ledger (user, purpose, model, tokens, cost, key kind house|byok) and alerts on drift > 5% (catches double-billing or missed logging). Intra-day checks (the chat budget pre-flight, doc 04) read today's ai_runs rows or the proxy's live spend endpoint — the ledger is the reconciled nightly view, not a live counter. Surfaced: month-to-date spend by purpose in Settings; per-run cost in Admin; alert at 80% of any budget.

As built (phase 8) — how ai_runs.cost_usd is actually captured, in priority order: the gateway's x-litellm-response-cost response header on non-streaming calls; an async spend-log settle for streaming chat (LiteLLM never attaches the cost header to a stream — the turn captures x-litellm-call-id and a post-commit task polls the proxy's own spend log and overwrites with the authoritative figure, never blocking the stream); the per-alias price-table estimate as last resort. The reconciler proved itself live: a chat turn persisted the Opus-class estimate at 49× the real Gemini spend and fired a genuine 95.8% drift alert, which is what motivated the settle path. Drift joins on metering identity only, key_kind comes from the call-time stamp (§2), and day bucketing is explicit UTC. Budgets as settled: $20/mo house cap · $10/mo per-user virtual key · $2/day chat · alert at 80%._

Expected steady-state (single user): story synthesis $2–5/mo (house) · daily generation $1–2/mo · chat usage-dependent, capped by budget · embeddings ≈ noise · infra: one small VPS $10–20/mo + object storage pennies.

4. Data isolation

Postgres RLS (Supabase): policies on every user-scoped table keyed to auth.uid(), shipped enabled from phase 0 in the same migrations that create the tables — with the anon key in the browser this is table stakes, not deferred hardening. The brain's service-role connection bypasses RLS; tenancy there is enforced in code via the user-context pattern, made structural: per-user repositories and lake handles can only be constructed from a user context — there is no per-call user_id parameter to forget — and CI carries cross-tenant tests (written alongside the phase-1 tables, not deferred to phase 8) that attempt another user's rows, prefixes, and tool calls through every repository and agent tool and must come back empty. Before multi-user launch, evaluate SET LOCAL app.user_id on brain transactions with matching defense-in-depth policies, so the database backstops the code path too. Phase 8 becomes a coverage review, not an enablement. A CI lint fails any migration adding a user_id table without a policy. As built (phase 8): the coverage review found and closed a real gap — the core phase-1 portfolio tables had zero cross-tenant probes (73 tests added). The RLS lint shipped with a code-frozen legacy allowlist (ai_runs/0005, usage_ledger/0010 — applied migrations are hash-immutable so they can't be annotated retroactively); new migrations must annotate deny-all in-file, and unrecognized CREATE TABLE forms fail loud. usage_ledger and eval_results are deny-all (the ai_runs precedent — Admin reads via brain endpoints).
Lake prefixes: the lake module's path builder is private; per-user reads/writes require a user context object and can only produce users/{that_id}/… paths. Shared market/… is read-only to per-user code paths.
Agent surface: tools receive user identity from the session, never as a model-visible parameter (established in designs 02/04) — isolation holds even if the model is prompt-injected by a malicious news story, which is also why story synthesis output is schema-constrained.
Traces/logs: spans reference fact/story/run IDs rather than payloads where practical; no keys, no full account numbers anywhere in logs. As built (phase 8): redaction is enforced at our own span-processor layer — scrubbing both attributes and exception events (a raw str(exc) exports Postgres constraint detail verbatim), and stripping the system prompt pydantic-ai leaks via model_request_parameters even with include_content=False; the OTel SDK is pinned with a canary test. Also closed here: FastAPI's default 422 echoed request bodies — including a pasted BYOK secret — back to the client; an app-wide validation-error handler strips input.

5. Secrets and key management

Dev: .env (gitignored) + MinIO/local defaults. Prod: SOPS-encrypted env or 1Password CLI injection on the single box. BYOK key custody is Supabase Vault's: the encryption key lives in Supabase's backend outside the database, so there is no master key for us to hold (fallback if Vault's status becomes a concern: envelope encryption in the brain with the key in VPS env — the decision doc keeps the schema identical either way). As built (phase 8): Vault custody is the shipped path (0.3.1 verified installed); the live E2E proved by SQL and log grep that a pasted secret exists nowhere but Vault — api_keys holds last4 only — and envelope encryption stays the documented fallback. Rotation runbook: rotate house provider keys by proxy config reload; rotate the Supabase service-role and anon keys (and DB password) from the dashboard — the brain's JWKS verification follows JWT signing-key rotation automatically via kid lookup.

6. Observability

OTel end-to-end: Pydantic AI agents, FastAPI, and pipeline jobs emit traces/metrics. Key metrics with alerts: pipeline failure/duration, quarantine counts, agent validation-retry rate and fallback events, daily cost by purpose, action acceptance rate, chat error rate. Alerts land in the app's admin Alerts view (+ optional email/ntfy).

As built (phase 8, decision): OTel wiring only, no hosted backend — Logfire is deferred, not adopted (self-hosted collector + Grafana/Tempo remains the documented exit either way). OTLP export is env-gated; ai_runs.trace_id is populated; alerts are computed brain-side into the notifications machinery and surface in the Admin Alerts view, so alerting works without any observability vendor.

7. Evals as CI

Prompts are versioned files in-repo; ai_runs.prompt_version references them; changing a prompt requires the eval gate.
Gate contents (from design 02 §5): golden fact packs → validator compliance + expected-action assertions + numeric-faithfulness extraction; scripted chat scenarios asserting tool-call sequences (what-if produces engine numbers, draft-action passes validators); small LLM-judge rubric for tone/prioritization, advisory not blocking.
Results recorded in eval_results, visible in Admin, trend-charted so quality regressions are seen, not felt.

As built (phase 8): CI did not exist before this phase — .github/workflows/ci.yml was written from scratch (brain suite on a Supabase-flavored Postgres image, node jobs, the RLS lint). The eval harnesses from phases 4–7 run as a six-prompt-version gate the recorder cannot weaken, and brain evals record writes eval_results rows.

8. Backups and recovery

Nightly pg_dump to versioned object storage (30-day retention) was the design here — superseded by a phase-8 user decision: Supabase Pro daily backups only (7-day retention), no self-owned pg_dump job. The lake bucket is versioned and raw is immutable — curated is rebuildable by design. LiteLLM's schema is config-plus-spend (restorable). Restore runbook (in-repo, docs/execution/runbooks/runbook-deploy-ec2.md): restore the Supabase backup → replay curated from raw if needed → verify with a facts re-run for the last trading day. RPO 24h, acceptable for personal use; the shorter retention is the accepted trade for zero backup infrastructure to operate.

9. Deployment topology

v1 (single box + Supabase): docker-compose on one small VPS or home server: Caddy (TLS + /brain/* proxy) → Next.js + FastAPI brain; LiteLLM; MinIO (or point straight at R2 and skip MinIO in prod); host cron invoking brain CLI. The app DB is Supabase — no Postgres container in prod compose; the supabase CLI stack stands in locally. Pipelines and the brain connect through Supavisor: session mode (port 5432) is the default for the long-running nightly chain and brain workers; transaction mode (6543) is acceptable for short bursty jobs but requires disabling the async driver's prepared-statement cache. Everything 12-factor.

As built (phase 8): the hosting target is AWS EC2 (t4g.small ARM primary) with the runbook written for it (docs/execution/runbooks/runbook-deploy-ec2.md); the prod compose, Caddyfile, and runbook are committed and validated — the actual EC2 stand-up is a follow-on operator session. Two Caddyfile facts worth knowing: chat must route to the web app before the /brain/* block (the app's bearer-attaching chat route handler would otherwise be shadowed and every prod chat call would 401 — caught end-to-end in review), and the public /brain/* surface is a deliberate, JWT-and-owner-gated operator convenience with the delete option spelled out — decide narrow-vs-keep at the deploy. The web image builds output: "standalone", and prod Supabase must use asymmetric (ES256/RS256) JWT signing — the brain refuses HS256, with a runbook preflight.

Scale-out sketch (when multi-user): already on managed Postgres — bump the Supabase compute tier as needed; R2 for the lake, brain + web on Fly/Railway, LiteLLM colocated with the brain, cron → Dagster. If the web tier lands on a serverless platform instead (e.g. Vercel), check function-duration limits on the same-origin /brain/* SSE proxy first — chat streams are long-lived, so pick a plan that allows long-running responses or stream straight from the brain's origin. No code changes — the seams (lake module, connector interface, gateway aliases, RLS) were built in from phase 0.

10. Open questions

All four were settled by the phase-8 decisions.

Resolved — observability backend: neither, yet — OTel wiring with no hosted backend (Logfire deferred); alerts are brain-computed notifications in the Admin Alerts view (§6).
Resolved — hosting target: AWS EC2 (t4g.small ARM primary), runbook committed; the stand-up itself is a follow-on operator session (§9).
Resolved — budgets: $20/mo house and $10/mo per-user confirmed as proposed; chat is $2/day (the phase-7 value, superseding $1/day); alert at 80% of any budget.
Resolved — backup encryption: moot — backups are Supabase Pro's daily backups (7-day retention) by user decision; there are no self-owned dumps to encrypt (§8).