Productized stacks with Hiroba primitives, KRO, and Argo CD

In the last post I covered how I build agnostic primitives — PostgresInstance, ObjectBucket — as thin cdk8s-generated XRDs reconciled by one Go composition function. Primitives are the stable contract: a team asks for a database, gets an endpoint and a secret reference in status, and never learns what sits behind it. This post is about the layer above — how I turn those primitives into a productized observability stack that teams can request as a single resource.

Why the stack doesn't live in the orchestrator

The first version of ObservabilityStack — Grafana, Loki, a metrics backend, and Alloy — was a handler in the same Go function as the primitives. The problem showed up the first time someone needed a Helm chart bump: that meant editing Go, rebuilding the function image, and rolling the control plane for what was really a targetRevision change. And every time a team wanted to tune a chart value, the stack's schema grew another escape hatch. Stacks are Helm orchestration. The orchestrator is a provider-branching engine. Forcing both into one binary made both worse.

So the stack moved to KRO — a ResourceGraphDefinition written in plain YAML — with Argo CD doing the actual chart installations. The Go function shrank to exactly two handlers, the primitives. That's the split I now enforce: primitives live in the Crossplane function because they need provider logic and dependency gating; stacks live in KRO because they're composition and GitOps, and status is the only contract between the two layers.

What the stack looks like

A team creates one resource and gets a working Grafana, Loki, Mimir (or Prometheus), and an in-cluster Alloy agent:

The RGD turns that into two kinds of resources. First, Hiroba primitives — an ObjectBucket for Loki's chunks, another for Mimir's blocks, a PostgresInstance for Grafana's database. Second, one Argo CD Application per chart, created in the team's namespace. KRO wires them together: the Loki Application's Helm values are built from the bucket's status.endpoint and bucket name, so the chart only gets created once the primitive reports where storage lives.

Each Application is multi-source, and that layout is the heart of the product design:

Source one is the chart with a valuesObject containing platform wiring — endpoints, bucket names, credential secret references. It always wins. Source two is the team's own repository, referenced as $values, where clients/team-api/observability/loki.yaml holds the things that are theirs: retention, resource limits, ingress annotations. Teams tune their stack in Git without ever touching the RGD, and they can't accidentally override the wiring that makes it work. Every chart also gets a fullnameOverride, which makes service names deterministic — the Grafana datasource ConfigMap, the Alloy remote-write URLs, and every cross-reference depend on names I can predict before anything is deployed.

Optional modules come from includeWhen. Disable Mimir and its Application and bucket vanish from the graph; switch metrics.backend to prometheus and a different Application with a different chart appears instead.

The remote agent

Client clusters get a second RGD, ObservabilityAgent: a lightweight Alloy daemonset that pushes logs and metrics to a management-plane stack over mTLS, with an X-Scope-OrgID header set to the team's tenant. Its River config — endpoints, tenant, certificate paths — was originally a single escaped CEL string inlined into the Helm values, which was unreadable in review. The version that survived emits the config as a dedicated ConfigMap inside the graph and points the chart at it:

Same wiring, but now the generated config is a file you can actually diff.

What broke along the way

Two failures, in the order my kind e2e found them — each taught me something real about KRO.

KRO readiness gates can't reference other resources. I wanted the Grafana Application to wait for its database to report Ready, and wrote that as an explicit readiness condition. KRO rejected the whole graph — ordering comes from references, not gates. Because the Application's values already read the database's status.endpoint, KRO's dependency graph was doing the waiting for me. I deleted the redundant conditions.

KRO status fields can't reference the instance itself. My first RGD tried to publish the stack's Grafana URL as status.endpoint, computed from the resource name. KRO rejected that too — status can only be built from other resources in the graph. The endpoint is deterministic anyway, so I dropped the status field and documented the naming pattern. A small contract loss, honestly noted.

Proof, not vibes

The kind e2e builds a cluster with Crossplane, KRO, and Argo CD, applies both RGDs, and requests a stack and an agent. It asserts the agent's ConfigMap contains the real remote endpoints, tenant header, and certificate paths; that the Application references that ConfigMap and mounts the right mTLS secret; and that the stack produces its child bucket and database resources. A lighter check runs on every change to stacks/: all RGDs must parse, and resource IDs must be unique. Both run in CI, so a broken RGD can't merge.

The tradeoffs

KRO's CEL is weaker than Go. I lost computed status fields, some validation I would have written as code, and stack-level readiness semantics — an RGD reports Ready when its resources exist; actual health is Argo CD's job now. Debugging also crosses three controllers instead of one, and KRO itself is another control-plane component to keep current.

What I got back: a chart bump is a YAML edit and a pull request, teams tune their own values in Git, and the orchestrator only contains code that genuinely needs a programming language. If a future stack ever needs real branching logic instead of CEL, that's the signal it belongs back in the function as a primitive — everything else stays out.