fix(bootstrap-kit): install vcluster CRDs + controller on Sovereign (gates Org → vCluster spawn) (#1624)

Pre-stages the upstream loft-sh vcluster Helm chart source on the
Sovereign cluster so the Organization controller
(core/controllers/organization) can render per-tenant
`helm.toolkit.fluxcd.io/v2 HelmRelease` CRs that reference
`chart.spec.sourceRef name=loft namespace=vcluster-system` (the
controller's defaults at core/controllers/organization/cmd/main.go).

Without this slot, every per-tenant vcluster HelmRelease the
Organization controller writes into the per-Org Gitea repo fails
Source reconcile with:

    HelmRepository.source.toolkit.fluxcd.io "loft" not found

→ no per-tenant vCluster is ever spawned → the Organization
controller's reconciliation loop blocks on tenant onboarding.
Convergence blocker #2 (vCluster source install on Sovereign).

Different layer from existing slots 54/58/59 (bp-dmz/mgmt/rtz-
vcluster): those bundle loft-sh/vcluster 0.20.0 as a Helm subchart
to ship Sovereign-tier DMZ/MGMT/RTZ vClusters in a single OCI
artifact; this slot registers a live Flux source CR so per-TENANT
vClusters can be spawned by the Organization controller at runtime.
The two paths are independent.

Changes:
  - platform/bp-vcluster-helmrepo/chart/  — new chart (no upstream
    subchart, same shape as bp-gateway-api); installs
    Namespace `vcluster-system` + HelmRepository `loft` pointing
    at https://charts.loft.sh.
  - platform/bp-vcluster-helmrepo/blueprint.yaml — Blueprint CR.
  - clusters/_template/bootstrap-kit/60-bp-vcluster-helmrepo.yaml
    — slot 60 (first free slot after the vCluster cohort 54/58/59);
    dependsOn: bp-flux only. Default-ON.
  - clusters/_template/bootstrap-kit/kustomization.yaml — wires
    slot 60 in.
  - scripts/expected-bootstrap-deps.yaml — declares slot 60 in
    the canonical dependency DAG (audit `check-bootstrap-deps.sh`
    passes).

Verification:
  - helm lint platform/bp-vcluster-helmrepo/chart/ — clean.
  - kubectl kustomize clusters/_template/bootstrap-kit/ — renders
    HelmRepository name=loft namespace=vcluster-system.
  - bash scripts/check-bootstrap-deps.sh — PASSED (Present: 49,
    Drift: 0, Cycles: 0).
  - TestBootstrapKit_TemplateClusterParses — PASS for slot 60.

Co-authored-by: hatiyildiz <hatice.yildiz@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

clusters/_template/bootstrap-kit/60-bp-vcluster-helmrepo.yaml

@@ -0,0 +1,109 @@
+# bp-vcluster-helmrepo — Catalyst bootstrap-kit Blueprint slot 60.
+#
+# Pre-stages the upstream loft-sh vcluster Helm chart source on the
+# Sovereign cluster so the Organization controller
+# (core/controllers/organization) can render per-tenant
+# `helm.toolkit.fluxcd.io/v2 HelmRelease` CRs whose `sourceRef` points
+# at `name=loft, namespace=vcluster-system` (the controller's defaults
+# at core/controllers/organization/cmd/main.go).
+#
+# Without this slot, every per-tenant vcluster HelmRelease the
+# Organization controller writes into the per-Org Gitea repo fails
+# Source reconcile with:
+#
+#   HelmRepository.source.toolkit.fluxcd.io "loft" not found
+#
+# → no per-tenant vCluster is ever spawned → the Organization
+# controller's reconciliation loop blocks on tenant onboarding.
+# Convergence blocker #2 (vCluster source install on Sovereign).
+#
+# Wrapper chart: platform/bp-vcluster-helmrepo/chart/
+#   Pure source-registration chart — registers a HelmRepository CR +
+#   the vcluster-system namespace it lives in. Ships NO upstream
+#   subchart (same shape as bp-gateway-api). The upstream chart is
+#   pulled per-tenant by Flux at HelmRelease reconcile time, NOT
+#   bundled into this slot's OCI artifact.
+#
+# Reconciled by: Flux on the new Sovereign's k3s control plane.
+#
+# Slot 60 chosen as the first free slot after the existing vCluster
+# cohort (54/58/59 — DMZ/MGMT/RTZ Sovereign-tier vClusters). This
+# slot is the per-TENANT vCluster source registration (a different
+# layer): the Sovereign-tier slots embed loft-sh/vcluster 0.20.0 as
+# a Helm subchart so they ship a single OCI artifact; this slot
+# registers a live `source.toolkit.fluxcd.io/HelmRepository` CR so
+# the Organization controller's per-tenant rendered HelmReleases
+# can resolve `chart.spec.sourceRef name=loft namespace=vcluster-
+# system` at reconcile time. The two paths are independent — this
+# slot does NOT depend on slots 54/58/59 (and vice versa).
+#
+# dependsOn:
+#   - bp-flux  — Flux's source-controller must be Ready so the
+#                HelmRepository CR is actually reconciled (otherwise
+#                the CR sits without artifacts and downstream Flux
+#                HelmReleases that reference it can't resolve).
+
+---
+apiVersion: source.toolkit.fluxcd.io/v1beta2
+kind: HelmRepository
+metadata:
+  name: bp-vcluster-helmrepo
+  namespace: flux-system
+spec:
+  type: oci
+  interval: 15m
+  url: oci://ghcr.io/openova-io
+  secretRef:
+    name: ghcr-pull
+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: bp-vcluster-helmrepo
+  namespace: flux-system
+  labels:
+    catalyst.openova.io/slot: "60"
+    catalyst.openova.io/tenant-spawn: vcluster
+spec:
+  interval: 15m
+  releaseName: vcluster-helmrepo
+  # The release marker Secret lives next to every other bootstrap-kit
+  # release. The chart's templates/namespace.yaml creates the actual
+  # vcluster-system namespace (cluster-scoped Namespace resource).
+  targetNamespace: flux-system
+  dependsOn:
+    - name: bp-flux
+  chart:
+    spec:
+      chart: bp-vcluster-helmrepo
+      version: 0.1.0
+      sourceRef:
+        kind: HelmRepository
+        name: bp-vcluster-helmrepo
+        namespace: flux-system
+  install:
+    timeout: 5m
+    disableWait: false
+    remediation:
+      retries: 3
+  upgrade:
+    timeout: 5m
+    disableWait: false
+    remediation:
+      retries: 3
+  # Per-Sovereign overlay surface — operators MAY swap the upstream URL
+  # for a Harbor proxy cache (MIRROR-EVERYTHING per
+  # docs/INVIOLABLE-PRINCIPLES.md #4a) or rename the CR / namespace
+  # to align with a custom Organization-controller config.
+  #
+  # Defaults match the controller's hardcoded defaults at
+  # core/controllers/organization/cmd/main.go:
+  #   CATALYST_VCLUSTER_HELMREPO_NAME      = "loft"
+  #   CATALYST_VCLUSTER_HELMREPO_NAMESPACE = "vcluster-system"
+  values:
+    vclusterHelmRepo:
+      name: loft
+      namespace: vcluster-system
+      url: https://charts.loft.sh
+      interval: 15m
+      createNamespace: true

clusters/_template/bootstrap-kit/kustomization.yaml

@@ -113,6 +113,17 @@ resources:
   # comment for the migration plan.
   - 58-bp-mgmt-vcluster.yaml
   - 59-bp-rtz-vcluster.yaml
+  # bp-vcluster-helmrepo (slot 60) — pre-stages the upstream loft-sh
+  # vcluster Helm chart source so the Organization controller
+  # (core/controllers/organization) can render per-tenant
+  # `helm.toolkit.fluxcd.io/v2 HelmRelease` CRs whose `chart.spec.
+  # sourceRef` points at `name=loft, namespace=vcluster-system`.
+  # Convergence blocker #2 (vCluster source install on Sovereign).
+  # Different layer from slots 54/58/59 (those bundle loft-sh/vcluster
+  # 0.20.0 as a subchart for the Sovereign-tier DMZ/MGMT/RTZ vClusters;
+  # this slot registers a live Flux source so per-TENANT vClusters can
+  # be spawned by the Organization controller at runtime). Default-ON.
+  - 60-bp-vcluster-helmrepo.yaml
   # bp-newapi (slot 80) — multi-tenant LLM marketplace gateway. Sequenced
   # after the W2.K1 dependency wave (cnpg/keycloak/openbao Ready) so
   # NewAPI's ExternalSecret + DSN dependencies resolve on first reconcile.

platform/bp-vcluster-helmrepo/blueprint.yaml

@@ -0,0 +1,46 @@
+apiVersion: catalyst.openova.io/v1alpha1
+kind: Blueprint
+metadata:
+  name: bp-vcluster-helmrepo
+  labels:
+    catalyst.openova.io/section: pts-3-1-networking-and-service-mesh
+    catalyst.openova.io/category: platform
+spec:
+  version: 0.1.0
+  card:
+    title: vCluster HelmRepository
+    summary: |
+      Pre-stages the upstream loft-sh vcluster Helm chart source on the
+      Sovereign cluster so the Organization controller
+      (core/controllers/organization) can render per-tenant vCluster
+      HelmReleases that reference `sourceRef name=loft namespace=
+      vcluster-system`. Convergence blocker #2 (vCluster source install
+      on Sovereign). Different layer from bp-mgmt/dmz/rtz-vcluster
+      (those bundle the upstream chart as a subchart for Sovereign-tier
+      vClusters; this slot registers a live Flux source so per-TENANT
+      vClusters can be spawned by the Organization controller at
+      runtime).
+    icon: vcluster.svg
+    category: platform
+  visibility: unlisted              # mandatory bootstrap topology
+  placementSchema:
+    modes: [primary-only]
+    minRegions: 1
+    maxRegions: 1
+  upgrades:
+    from: ["0.x"]
+  manifests:
+    chart: ./chart
+  depends:
+    - blueprint: bp-flux
+      version: ^2
+
+  # ── Outputs advertised to dependent Blueprints ───────────────────────────
+  # The Organization controller's per-tenant vCluster renderer
+  # (core/controllers/organization/internal/gitops/manifests.go)
+  # references these via env (CATALYST_VCLUSTER_HELMREPO_NAME +
+  # CATALYST_VCLUSTER_HELMREPO_NAMESPACE).
+  outputs:
+    helmRepoName: loft
+    helmRepoNamespace: vcluster-system
+    upstreamURL: https://charts.loft.sh

platform/bp-vcluster-helmrepo/chart/Chart.yaml

@@ -0,0 +1,64 @@
+apiVersion: v2
+name: bp-vcluster-helmrepo
+# 0.1.0 — initial implementation. Pre-stages the upstream loft-sh
+# vcluster Helm chart source on the Sovereign cluster so the
+# Organization controller (core/controllers/organization) can render
+# per-tenant `helm.toolkit.fluxcd.io/v2 HelmRelease` CRs whose
+# `chart.spec.sourceRef` points at `name=loft, namespace=vcluster-system`
+# (see core/controllers/organization/internal/gitops/manifests.go,
+# defaults at core/controllers/organization/cmd/main.go).
+#
+# Without this slot, every per-tenant vcluster HelmRelease that the
+# Organization controller writes into the per-Org Gitea repo fails
+# Source reconcile with "HelmRepository.source.toolkit.fluxcd.io
+# \"loft\" not found" — convergence blocker #2 (vCluster CRD-equivalent
+# install on Sovereign). Spawning any tenant org's vCluster gates on
+# this slot.
+version: 0.1.0
+appVersion: "0.33.0"
+description: |
+  Catalyst Blueprint that pre-stages the upstream loft-sh vcluster
+  Helm chart source on the Sovereign cluster.
+
+  Creates:
+    1. Namespace `vcluster-system` (configurable) — holds the
+       HelmRepository CR and any future loft-sh tooling.
+    2. HelmRepository `loft` in that namespace pointing at
+       https://charts.loft.sh — the canonical upstream chart repo.
+
+  Why a separate Blueprint instead of a subchart bundle:
+
+    Per-tenant vClusters spawned by the Organization controller
+    (core/controllers/organization) are rendered as
+    `helm.toolkit.fluxcd.io/v2 HelmRelease` CRs in per-Org Gitea
+    repos. Flux on the Sovereign reconciles them. Each HelmRelease
+    references `sourceRef.kind=HelmRepository name=loft
+    namespace=vcluster-system` to pull the upstream chart.
+
+    The Sovereign-tier wrapper charts (bp-mgmt-vcluster,
+    bp-dmz-vcluster, bp-rtz-vcluster) bundle loft-sh/vcluster
+    0.20.0 as a Helm subchart so they ship a single OCI artifact —
+    that path does NOT register a HelmRepository the controller can
+    use. Per-tenant vClusters need a live `source.toolkit.fluxcd.io/
+    HelmRepository` CR on the cluster, not an embedded subchart, so
+    Flux can resolve the per-Org HelmRelease independently of any
+    bundled subchart.
+
+    Same shape as bp-gateway-api: legitimately ships NO upstream
+    subchart — the deliverable IS the Flux source CR registration.
+
+  Reconciled by Flux as bootstrap-kit slot 45 (between
+  cluster-autoscaler and the per-region vCluster slots so the source
+  is ready before any HelmRelease that references it lands).
+type: application
+annotations:
+  # CI Blueprint Release smoke-render gate — this chart's templates
+  # render unconditionally (no top-level enable gate; the Sovereign
+  # always needs the loft HelmRepository to spawn tenant vClusters).
+  catalyst.openova.io/smoke-render-mode: "default-on"
+  catalyst.openova.io/no-upstream: "true"
+  catalyst.openova.io/upstream-loft-vcluster-chart-version: "0.33.x"
+keywords: [catalyst, blueprint, vcluster, loft, helmrepository, tenant-spawn]
+maintainers:
+  - name: OpenOva Catalyst
+    email: catalyst@openova.io

platform/bp-vcluster-helmrepo/chart/templates/_helpers.tpl

@@ -0,0 +1,13 @@
+{{/*
+Common labels for every resource the chart renders. Per
+docs/EPICS-1-6.md §1.1 every Catalyst-managed resource carries the
+canonical label set.
+*/}}
+{{- define "bp-vcluster-helmrepo.labels" -}}
+app.kubernetes.io/name: {{ .Chart.Name }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
+catalyst.openova.io/blueprint: {{ .Chart.Name }}
+{{- end }}

platform/bp-vcluster-helmrepo/chart/templates/helmrepository.yaml

@@ -0,0 +1,23 @@
+# HelmRepository CR registering the upstream loft-sh chart repo as a
+# Flux source. Consumed by the Organization controller's per-tenant
+# HelmRelease render path (see
+# core/controllers/organization/internal/gitops/manifests.go:96-100).
+#
+# Apply order: Flux dependsOn (slot 45 has no chart-internal deps, but
+# the bootstrap-kit slot file's `dependsOn: bp-flux` keeps Flux's
+# source-controller up before this lands).
+apiVersion: source.toolkit.fluxcd.io/v1beta2
+kind: HelmRepository
+metadata:
+  name: {{ .Values.vclusterHelmRepo.name | quote }}
+  namespace: {{ .Values.vclusterHelmRepo.namespace | quote }}
+  labels:
+    {{- include "bp-vcluster-helmrepo.labels" . | nindent 4 }}
+    catalyst.openova.io/tenant-spawn: vcluster
+spec:
+  # `type` omitted on purpose — defaults to the traditional
+  # index.yaml-style HelmRepository (loft.sh's chart server speaks
+  # this protocol). Don't set `type: oci` here — charts.loft.sh is NOT
+  # an OCI registry.
+  interval: {{ .Values.vclusterHelmRepo.interval | quote }}
+  url: {{ .Values.vclusterHelmRepo.url | quote }}

platform/bp-vcluster-helmrepo/chart/templates/namespace.yaml

@@ -0,0 +1,20 @@
+{{- if .Values.vclusterHelmRepo.createNamespace -}}
+# Namespace that hosts the upstream loft-sh vcluster Helm chart source
+# CR. The Organization controller's rendered per-tenant HelmReleases
+# reference this namespace via `chart.spec.sourceRef.namespace` (see
+# core/controllers/organization/internal/gitops/manifests.go default
+# `VClusterHelmRepoNamespace = "vcluster-system"`).
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: {{ .Values.vclusterHelmRepo.namespace | quote }}
+  labels:
+    {{- include "bp-vcluster-helmrepo.labels" . | nindent 4 }}
+    # pod-security: restricted — this namespace only ever hosts Flux
+    # source CRs and downstream Helm release marker Secrets; no
+    # workload Pods land here. The actual vCluster StatefulSet lives
+    # in the per-tenant Org namespace, not here.
+    pod-security.kubernetes.io/enforce: restricted
+    pod-security.kubernetes.io/audit: restricted
+    pod-security.kubernetes.io/warn: restricted
+{{- end }}

platform/bp-vcluster-helmrepo/chart/values.yaml

@@ -0,0 +1,36 @@
+# Catalyst Blueprint values for bp-vcluster-helmrepo.
+#
+# Pre-stages the upstream loft-sh vcluster Helm chart source on the
+# Sovereign cluster. Consumed by the Organization controller
+# (core/controllers/organization) when spawning per-tenant vClusters.
+#
+# Per docs/INVIOLABLE-PRINCIPLES.md #4 (never hardcode) every value
+# below is operator-overridable per-Sovereign.
+
+vclusterHelmRepo:
+  # The HelmRepository CR's namespace. Must match the value
+  # CATALYST_VCLUSTER_HELMREPO_NAMESPACE that the Organization
+  # controller is configured with (defaults to "vcluster-system" at
+  # core/controllers/organization/cmd/main.go).
+  namespace: "vcluster-system"
+
+  # The HelmRepository CR's name. Must match the value
+  # CATALYST_VCLUSTER_HELMREPO_NAME the Organization controller is
+  # configured with (defaults to "loft").
+  name: "loft"
+
+  # Upstream loft-sh chart repo URL. Per docs/INVIOLABLE-PRINCIPLES.md
+  # #4a (SHA-pinned / mirror-everything) operators MAY override this to
+  # point at their Harbor proxy cache
+  # (e.g. "https://harbor.openova.io/chartrepo/proxy-loft").
+  url: "https://charts.loft.sh"
+
+  # Source-controller reconcile interval. 15m matches every other
+  # bootstrap-kit HelmRepository (bp-cilium / bp-cert-manager /
+  # bp-gateway-api).
+  interval: "15m"
+
+  # Whether to create the namespace. Most Sovereigns will want this on;
+  # operator-MAY turn it off to manage the namespace via a separate
+  # Kustomization.
+  createNamespace: true

scripts/expected-bootstrap-deps.yaml

@@ -444,6 +444,25 @@ slots:
       - bp-cert-manager
     wave: present
 
+  # ---- Slot 60 — bp-vcluster-helmrepo (convergence blocker #2 —
+  # per-tenant vCluster source install on Sovereign).
+  # Pre-stages the upstream loft-sh vcluster Helm chart source on the
+  # Sovereign cluster so the Organization controller
+  # (core/controllers/organization) can render per-tenant
+  # `helm.toolkit.fluxcd.io/v2 HelmRelease` CRs that reference
+  # `chart.spec.sourceRef name=loft namespace=vcluster-system`. Without
+  # this slot, every per-tenant vcluster HelmRelease fails Source
+  # reconcile with "HelmRepository.source.toolkit.fluxcd.io \"loft\"
+  # not found" → no tenant vCluster is ever spawned. Different layer
+  # from slots 54/58/59 which bundle the upstream chart as a subchart
+  # for the Sovereign-tier DMZ/MGMT/RTZ vClusters. Default-ON; ships
+  # NO upstream subchart (same shape as bp-gateway-api).
+  - slot: 60
+    name: bp-vcluster-helmrepo
+    depends_on:
+      - bp-flux
+    wave: present
+
   # ---- Slot 80 — bp-newapi multi-tenant LLM marketplace gateway. Issue #799.
   # Sequenced past the W2.K4 numbering plan (slots 36-48) so it never
   # collides with the AI-runtime / observability / livekit cohort. The