GREEN PACKAGE SKILL · DIGITALOCEAN · RYBBIT

Rybbit Package Skill

A reproducible Green workflow for one production-oriented Rybbit analytics platform: PostgreSQL 17 for metadata and authentication, ClickHouse for columnar events, Redis, and the Rybbit backend and client behind Caddy, with Cloudflare DNS and TLS.

ClickHouse is backed up with its own BACKUP statement. A hot tar of the data directory races the server’s merges and produces an archive that cannot be restored. This package takes a consistent native backup, verifies the PostgreSQL dump restores, and only then uploads — a failed backup fails the unit rather than leaving a plausible-looking object in the bucket.

Quick start

npx skills add getcolors/rybbit
cp .agents/skills/package-rybbit-green/green ./green
chmod +x green
./green build
./green create --dry-run

build renders the deployment without credentials. The dry-run walks the whole workflow while skipping every side effect. A real create or delete touches live infrastructure and requires separate authorization.

Architecture and sizing

Compute

OpenTofu discovers the configured region’s default VPC at plan time and creates one Ubuntu 24.04 Droplet plus a firewall. No VPC is created and no VPC UUID belongs in desired state.

Runtime

Six containers: postgres:17-alpine, clickhouse-server:24.8-alpine, redis:8.6.4-alpine, the Rybbit backend and client, and caddy:2.11.4. Every datastore stays on the private Compose network.

Public boundary

Cloudflare publishes one A record. Caddy routes /api/* and the OAuth discovery paths to backend:3001 and everything else to client:3002. Only 22, 80 and 443 are admitted.

Sizing

s-4vcpu-8gb. ClickHouse and PostgreSQL share the host with two Node processes, so this is the practical floor rather than a generous allocation.

Secrets for the datastores are generated on the host on first converge into /opt/rybbit/stack.env at mode 0600 and retained across runs, so a re-converge never rotates a password out from under a running database. This is a single-node design: process and host restarts recover, but a Droplet, disk, or regional failure causes downtime.

Desired state and credentials

colors.yml is the only file you edit, and it holds non-secret values only: the profile, DigitalOcean region, size and SSH sources, the host, every image tag, the data directories, and the backup destination, schedule and retention. See the configuration reference for every key.

Variable	Purpose
`COLORS_PAR_DO_TOKEN`	DigitalOcean compute and firewall
`COLORS_PAR_CLOUDFLARE_API_TOKEN`	DNS record for the configured host
`COLORS_PAR_RYBBIT_BACKUP_R2_ACCESS_KEY_ID` `COLORS_PAR_RYBBIT_BACKUP_R2_SECRET_ACCESS_KEY`	R2 credentials for the backup bucket

Database, ClickHouse, Redis and auth secrets are not supplied here — they are generated on the host and never leave it.

rybbit-disable-signup is desired state. Rybbit has no first-run bootstrap: with signup disabled nobody can create the first account, so leave it false until you have registered, then set it true. It is written into stack.env on first converge only.
Never export COLORS_PAR_PROFILE. The profile keys remote state.
Never edit or commit generated .colors/ output.
Keep compute-prevent-destroy: true in committed desired state.

Acceptance and operations

./green create
ssh <droplet> 'cd /opt/rybbit && docker compose ps'
ssh <droplet> 'systemctl status rybbit-backup.timer'
ssh <droplet> 'journalctl -u rybbit-backup.service --since=-24h'

Acceptance runs at the end of create and reports only what it checked:

HTTPS with certificate verification. /api/health must answer over a valid certificate.
Ingestion is read back from ClickHouse. A synthetic pageview is posted to /api/track and the events table must grow. A 2xx with no stored row is reported as dropped, not success; with no site configured the verdict is not-configured, which passes and claims nothing. The events table is resolved from system.tables rather than hardcoded, so a schema change fails loudly instead of silently checking nothing.
The backup drill is confirmed in R2. The unit is started, the timer must be active, and the profile prefix must then contain a non-empty object newer than the run.

Backups, restoration, and disaster recovery

A systemd timer runs nightly at 02:30 UTC. Each run dumps PostgreSQL, restores that dump into a scratch database and requires the schema back, takes a native ClickHouse BACKUP into the server’s allowed path on the bind mount, consolidates both into one archive, and uploads it to R2 under the profile prefix. There is deliberately no tar fallback for ClickHouse: an archive that cannot be restored is worse than a failed unit.

Retention applies to the bucket and the disk, both at rybbit-backup-retention-days.

To recover, unpack the newest archive, restore postgres.sql into an empty PostgreSQL 17 database and clickhouse.zip with ClickHouse RESTORE, then verify the dashboard and event ingestion before moving DNS. Never test restoration over live data.

Prevent-destroy is not a backup. Deleting compute does not authorize deleting R2 objects, the discovered default VPC, unrelated DNS, or any other provider resource. The backend and client images are pinned by digest, so a converge cannot pick up a new upstream build without a change here.

Development and source

bb test
bb golden
bb golden:accept
./scripts/launcher.sh

Golden fixtures render the whole deployment from a fixture colors.yml and compare byte for byte; the accept step refuses to copy a rendered tree containing a credential-shaped value. Source on GitHub. Read every golden diff — never accept generated output merely to make the suite pass.