Slice F layers three concerns on top of K-Cont-2's reconciler +
sequencer:
F-1 — extend audit-emit coverage with three new audit-types:
- continuum-cr-created — fires once per CR observation
- continuum-config-changed — fires on switchover-relevant spec drift
- continuum-lease-collision — fires when Acquire returns
ErrLeaseHeldByAnother during the
opportunistic re-acquire path
Total reserved Continuum audit-types now 12 (was 9). Order is
K-Cont-2's 9 first, then F-1's 3 (additions at end so existing
index-pinned tests keep working). U-DR-1 subscribes by
audit-type=continuum-* so it receives the new types automatically.
F-2 — Sequencer.DryRun + DryRunReport struct + per-step
preconditions evaluator. Walks the same 7 steps Execute would run,
but read-only end-to-end (asserted by tests: zero audit emits, zero
state mutation). Per-step durations as exported constants. Plan
content fingerprint (16-hex SHA-256 prefix) for cache idempotency.
Blockers (FATAL) vs Warnings (advisory) so the UI can render the
report and disable [ Confirm Switchover ] when blockers present.
F-3 — Sequencer.PostSwitchoverHealth + HealthReport struct + 4
fixed-order checks (replicas-healthy, dns-probes, latency-normal,
audit-posted). Replicas check reads both halves of the cluster-pair
post-switchover (new-primary has replica.enabled=false; new-replica
has replica.enabled=true; both must be Ready=true). DNS check
fans out to multi-vantage resolvers (default 8.8.8.8 / 1.1.1.1 /
9.9.9.9) and asserts every (hostname × vantage) returns at least one
ToRegion IP. Latency check is permanently Deferred=true (Cilium
hubble metrics scrape is SRE follow-up). Audit check queries an
injected AuditTail (recorder in tests; NATS PullConsumer wiring is
follow-up — currently Deferred=true in production).
Controller chains PostSwitchoverHealth ~30s after every successful
switchover (HealthDelay; CONTINUUM_HEALTH_DELAY_SECONDS env). Result
written to Continuum CR status condition LastSwitchoverHealthy with
True/False/Unknown + one-line summary message.
Endpoints — small HTTP server in continuum-controller binary on
:8082 (CONTINUUM_API_ADDR env; empty disables):
- POST /v1/continuums/{ns}/{name}/dry-run → DryRunReport
- GET /v1/continuums/{ns}/{name}/health → HealthReport
- GET /healthz → ok
Auth — owner-tier gated per INVIOLABLE-PRINCIPLES #5:
X-Catalyst-Owner-Tier: true header (catalyst-api stamps it after JWT
validation) plus optional Authorization: Bearer <CONTINUUM_API_TOKEN>
for defence in depth. The /api/v1/sovereigns/{id}/... outer envelope
is the catalyst-api's responsibility (separate slice); the controller
exposes only the inner shape.
Chart — values.yaml + deployment.yaml + service.yaml extended with
continuum.api.{port,tokenSecretRef} and
continuum.health.postSwitchoverDelaySeconds. Service exposes new
api port (default 8082) so the catalyst-api proxy can reach it.
Tests — three-tier gate per implementer-canon §6:
- 53 unit tests across switchover (DryRun + Health + integration),
events (3 new types + roundtrip), api (server + auth + cache),
controller (4 new test groups for F-1 + F-3 chain).
- End-to-end integration test: DryRun → Execute → PostSwitchoverHealth
sequence (TestEndToEnd_DryRunThenSwitchoverThenHealth +
TestEndToEnd_DryRunBlockedSwitchoverNeverRuns).
- go test -count=1 -race ./... clean across all sibling controllers.
- go vet ./... clean.
K-Cont-2's sequencer surface was sufficient — this slice ADDED
DryRun + PostSwitchoverHealth methods without modifying the existing
Execute / RequestFailback / steps() implementations.
Out of scope (per slice F brief): WitnessClient interface changes,
CF Worker changes, U-DR-1 UI, 1M-row C-DB-3 acceptance test,
Cilium hubble latency metrics, NATS PullConsumer for audit-posted
health check (deferred).
Co-authored-by: hatiyildiz <hati.yildiz@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
96f8b260c9