Redis, under
reconciliation.

Green, Red and Blue

Choose the Clojure, TypeScript or Python Package Skill. Each provides build, create, check, rehearse, drill, restart and delete with the same desired state and ownership guards. The native controllers use their matching Kubernetes SDK and Redis workflow.

npx skills add getcolors/redis-operator --skill package-redis-operator-red
npx skills add getcolors/redis-operator --skill package-redis-operator-blue

Copy the installed red or blue launcher to your deployment root. Build the controller with scripts/image.sh REGISTRY SHA red or blue, then pin its digest in colors.yml. Any launcher can probe any controller colour. Changing controller language runs one convergence to record its configuration.

Red source · Blue source

One repository. Two execution boundaries.

The package installs the control plane. The controller keeps one external Redis Droplet converged per RedisDeployment.

(let [transport (client/kubectl-client (if (= context "--in-cluster")
                                         {:in-cluster? true} {:context context}))
        runtime (k8s/start! (k8s/controller {:packages [(redis/package)] :client transport
                                           :namespace (or namespace "colors-dev")
                                           :workers 1 :poll-ms 2000}))
        stopped (promise)]

Start with a render.

Build and create --dry-run work without credentials or cluster access. A real create installs infrastructure and requires explicit authorization.

# Install the package into a deployment project
npx skills add getcolors/redis-operator --skill package-redis-operator-green
cp .agents/skills/package-redis-operator-green/green green

# After filling in a non-secret colors.yml
./green build
./green create --dry-run

That installs the package-redis-operator-green Package Skill and copies its launcher to the deployment root; the launcher answers seven verbs — build, create, check, rehearse, drill, restart and delete — detailed in the command reference.

Provide an existing Kubernetes cluster, a usable KUBECONFIG, its explicit kube-context, a controller image digest, externally managed R2 state and backup buckets, and public worker/developer IPv4 /32 SSH sources. The launcher needs Babashka; live operations also use kubectl. The controller image contains the provisioning toolchain.

# Once credentials are present and creation is authorized
./green create
./green check

# Diagnostic reads: substitute your actual context and resource
kubectl --context <kube-context> get rd -n colors-redis
kubectl --context <kube-context> logs -n colors-redis   deployment/colors-redis-operator

build writes two files under .colors/<profile>/operator/: manifests.json (Namespace, CRD, ServiceAccount, Role, RoleBinding, PVC, Deployment) and redis-deployment.json. Credentials are applied separately on stdin. Dry-run skips the live steps but still renders these files.

Observe first. Converge when needed.

Existence, desired-state match and live service health are independent observations. Unknown state is never permission to recreate.

(defn provider-matches? [opts node droplet]
  (and (= (:name node) (:name droplet))
       (= (:digitalocean-region opts) (get-in droplet [:region :slug]))
       (= (:digitalocean-size opts) (:size_slug droplet))
       (= (:digitalocean-image opts) (get-in droplet [:image :slug]))
       (some #(and (= "public" (:type %)) (= (:ip node) (:ip_address %)))
             (get-in droplet [:networks :v4]))))

A successful convergence writes operator-success.edn with a SHA-256 configuration hash and the provider ID. The hash excludes event and resource metadata. An unchanged hash alone is insufficient: provider identity, region, size, image, recorded public IP and authenticated Redis PING must also match.

(defn config-hash [config]
  (let [desired (dissoc config :green/event :green.kubernetes/resource)]
    (apply str (map #(format "%02x" (bit-and 255 %))
                    (.digest (MessageDigest/getInstance "SHA-256")
                             (.getBytes (pr-str (into (sorted-map) desired)) "UTF-8"))))))

converge calls the Redis package workflow with :green/event :create, then re-reads owned state before persisting the success marker. Healthy periodic checks skip that workflow. Partial initial compute state requests convergence so the compute library can resume its ownership protocol.

Desired state is a contract.

A flat, non-secret colors.yml becomes spec.config plus the operator’s deployment settings. Validation reports all errors together with exit code 2.

# Selected fields; this is not a complete deployable configuration.
profile: redis-operator-example
kube-context: do-ams3-doks-fixture
namespace: colors-redis
reconcile-interval: 60s
deletion-policy: Retain
compute-prevent-destroy: true
provider-compute: digitalocean
provider-backend: r2

The API is colors.getcolors.ai/v1alpha1, kind RedisDeployment, short name rd, with namespace scope and a status subresource. Only state: running is supported. The CRD freezes profile presence and value, r2-bucket and r2-endpoint. Together, the backend endpoint, bucket and resolved profile identify ownership.

(defn resource
  "The RedisDeployment. `spec.suspend` is deliberately absent: the CRD
  defaults it on creation and an apply must never reset a suspension."
  [opts]
  {:apiVersion "colors.getcolors.ai/v1alpha1" :kind "RedisDeployment"
   :metadata {:name (:resource-name opts) :namespace (:namespace opts)}
   :spec {:state "running"
          :deletionPolicy (:deletion-policy opts)
          :reconcileInterval (:reconcile-interval opts)
          :config (validate/config opts)}})

Credentials enter at runtime.

Only five COLORS_PAR_* variables enter the controller. The controller rejects other configuration overrides at startup.

COLORS_PAR_DO_TOKEN
COLORS_PAR_R2_ACCESS_KEY_ID
COLORS_PAR_R2_SECRET_ACCESS_KEY
COLORS_PAR_REDIS_BACKUP_R2_ACCESS_KEY_ID
COLORS_PAR_REDIS_BACKUP_R2_SECRET_ACCESS_KEY

(defn check-environment! [env]
  (when (some #(and (str/starts-with? % "COLORS_PAR_")
                    (not (contains? credential-vars %))) (keys env))
    (throw (ex-info "Unexpected COLORS_PAR environment override; desired configuration must come from the resource" {})))
  (when (some #(str/blank? (get env %)) (keys credential-vars))
    (throw (ex-info "Required operator credentials are missing" {}))))

The package reads the process environment; it does not parse .envrc.private. KUBECONFIG selects the kubeconfig and every launcher-side kubectl call supplies --context. The probe’s internal kubectl call runs inside the Pod using the in-cluster environment.

The adapter forces DigitalOcean compute, R2 backend, unmanaged Redis storage buckets and compute-prevent-destroy: true during convergence. Work defaults to /data/work (COLORS_WORKDIR can override that root). Deletion temporarily lifts the guard only after the resource explicitly selects Destroy.

Seven verbs. Explicit evidence.

Operational success is proved against the resource generation, cloud identity and live Redis service.

Ready is polled, not sampled once

check and operational preconditions poll every 5 seconds for up to 180 seconds. Transient Reconciling and unobserved generations are allowed. Failed, Invalid, Blocked, suspension and deletion end normal readiness waits. Create tolerates Failed while the controller retries; the drill recovery loop also continues through failed reconciliation passes.

(defn ready?
  "Ready at the current generation: the controller has observed this spec and
  its Ready condition is True."
  [cr]
  (let [generation (get-in cr [:metadata :generation])
        status (:status cr)]
    (and (some? generation)
         (= generation (:observedGeneration status))
         (= "Ready" (:phase status))
         (boolean (some #(and (= "Ready" (:type %)) (= "True" (:status %)))
                        (:conditions status))))))

The probe crosses the boundary

Before any probe exec, the launcher waits for exactly one non-terminating Running controller Pod whose logs since its start time include RedisDeployment controller running. This avoids racing the controller’s dependency downloads. It then execs bb -m colors.probe inside the running controller. The probe reads the CR and owned compute state, then SSHes to the recorded node. Redis authentication is read inside the Redis container and placed in REDISCLI_AUTH, keeping passwords out of command arguments. Marker keys and values are constrained to a safe token alphabet.

(defn safe-token [value]
  (when-not (and (string? value) (re-matches #"[A-Za-z0-9:_-]{1,160}" value))
    (throw (ex-info "Invalid probe token" {}))) value)

Three different recovery proofs.

Backup rehearsal, machine replacement and controller restart answer different operational questions.

A. Rehearse a backup

(defn patch-resource!
  "A JSON patch guarded by a resourceVersion test, so a concurrent edit fails
  the operation instead of being overwritten."
  [opts current patch]
  (kubectl opts ["patch" resource-type (:resource-name opts) "-n" (:namespace opts)
                 "--type=json" "-o" "json" "-p"
                 (json/generate-string
                  (into [{:op "test" :path "/metadata/resourceVersion"
                          :value (get-in current [:metadata :resourceVersion])}]
                        patch))]
           {:json? true}))

Rehearsal resumes only when UID and suspension generation still match and remote completion is certain. A completed but failed rehearsal can safely resume while reporting failure. An uncertain exec, changed resource or unreadable re-check prevents automatic resume. The original error is preserved.

B. Delete a Droplet and prove self-healing

COLORS_PAR_DRILL_DELETE_OWNED_DROPLET=true ./green drill

The live ID must match recorded state. Droplet name, probe name and profile must be identical. The recorded IP must appear among public IPv4 addresses. Any k8s: tag rejects the target; with doks-cluster-id, node-pool Droplet IDs are explicitly excluded too. Probe/JSON decode failures during recovery are retried. Marker survival is recorded, but is not required for the service-recovery proof.

C. Restart the controller, preserve the Droplet

./green restart

The command requires a nonblank new Pod start time and rejects the old Pod UID if still present. Success also requires unchanged resource UID, generation, provider ID and convergence-record modification time, plus live health. If the old Pod cannot be confirmed stopped, the command intentionally leaves replicas at zero. It never force-deletes that Pod.

Finalization comes before teardown.

Retain is the resource default. The package delete command explicitly selects Destroy after its separate guard is lifted.

COLORS_PAR_COMPUTE_PREVENT_DESTROY=false ./green delete

The controller finalizer is colors.getcolors.ai/infrastructure. While it runs, the controller and its credentials must remain available. A confirmed missing Droplet skips unreachable application cleanup, but still performs local SSH configuration cleanup and removes owned shared compute resources.

The namespace deletion removes the controller installation and namespaced objects. Use a dedicated namespace for this installation. The committed compute-prevent-destroy stays true; even delete --dry-run refuses with exit 2 until its guard is lifted.

Follow the decision. Retain the detail.

Controller stdout explains each observe, converge and delete outcome. Detailed workflow failures stay on the persistent volume.

./green check

kubectl --context <kube-context> logs -n colors-redis   deployment/colors-redis-operator

kubectl --context <kube-context> exec -n colors-redis   deployment/colors-redis-operator --   ls -1 /data/work/<profile>/failures

kubectl --context <kube-context> exec -n colors-redis   deployment/colors-redis-operator --   cat /data/work/<profile>/failures/<file>

Failure reports retain the failed step, exit code, workflow error, optional Ansible recap and trace. Exact values of every nonempty COLORS_PAR_* environment variable are replaced with ***, longest first. Reports are atomically written as <UTC timestamp>-<step>.log, with mode 0600 in a 0700 directory; only the newest 20 remain.

(defn mask
  "Replace the exact value of every COLORS_PAR_* variable in `env` with ***,
  longest values first so a value that contains another is masked whole."
  [text env]
  (reduce (fn [text value] (str/replace text value "***"))
          (str text)
          (->> env
               (filter (fn [[k v]] (and (str/starts-with? (str k) "COLORS_PAR_") (not (str/blank? v)))))
               (map second)
               distinct
               (sort-by count >))))

Read, test, pin, release.

The package library and the controller image share source history, but deployments consume two independent pins.

bb test
bb golden
./scripts/launcher.sh
bb -e "(require 'colors.main 'colors.probe) (println :controller-loads)"

# Run the controller locally with explicit context and namespace
bb controller <context> <namespace>

The tests cover observation, convergence markers, retained diagnostics, probe suspension, validation, rendering, transports and lifecycle invariants. Golden fixtures protect the CRD and rendered installation, including RBAC, security context, pull Secret and the 10,800-second shutdown grace period. The launcher test copies the payload into a temporary standalone deployment and verifies credential-free renders and refusal guards.

# After an authorized, clean, pushed package revision and bb pin
npx skills update -p -y
cp .agents/skills/package-redis-operator-green/green green

For local work across repositories, use REDIS_OPERATOR_LIB_ROOT, GREEN_LIB_ROOT, REDIS_LIB_ROOT and COLORS_COMPUTE_LIB_ROOT; do not hand-edit pins. The package repository’s root launcher is a symlink, while a deployment’s installed launcher is a copy.

scripts/image.sh is the image build/push entry point. It uses DOCKER_CONFIG, targets linux/amd64, tags by source SHA, labels the revision and prints the immutable digest. The Dockerfile installs Babashka, kubectl, OpenTofu, Ansible, SSH, Redis CLI, AWS CLI, Git and Java. First startup and cache misses resolve pinned dependencies and need outbound access to GitHub and Maven repositories. The PVC also mounts /root/.gitlibs, /root/.m2, /root/.deps.clj and /app/.cpcache, preserving dependency caches across Pod restarts.

The source behind the manual.

Implementation excerpts are copied from this checkout. Diagrams and operational descriptions explain those exact contracts.

Source revision e59a5abab935035bf466a5c7159c195337e5c2ee. The original Green source links below remain pinned to this revision. The runtime choices section links to the current Red and Blue implementations. Pinned dependency internals are outside this repository’s source-reading scope. Source links point to this revision; no live infrastructure status is implied.