GREEN PACKAGE SKILL · DIGITALOCEAN · POSTHOG

PostHog Package Skill

A reproducible Green workflow for one single-node PostHog product analytics suite: the Django application and Celery worker, ClickHouse with embedded Keeper, Redpanda, Temporal, the Rust capture service and the Node plugin server, behind Caddy with Cloudflare DNS and TLS.

Ten containers, and none of them are optional. PostHog’s ingestion path is capture → Kafka → plugin server → ClickHouse, its ClickHouse schema is built from ReplicatedMergeTree tables that require Keeper, and its Django startup connects to Temporal. Upstream’s own single-server compose defines roughly 35 services; this is a deliberate reduction to the set that actually has to be present.

Quick start

npx skills add getcolors/posthog
cp .agents/skills/package-posthog-green/green ./green
chmod +x green
./green build
./green create --dry-run

build renders the deployment without credentials. The dry-run walks the whole workflow while skipping every side effect. A real create or delete touches live infrastructure and requires separate authorization.

Architecture and components

Compute

OpenTofu discovers the configured region’s default VPC at plan time and creates one Ubuntu 24.04 Droplet plus a firewall. No VPC is created and no VPC UUID belongs in desired state.

Public boundary

Cloudflare publishes one proxied A record. Caddy sends /capture, /e, /batch, /i/*, /track, /engage and /s to the capture service, and everything else to the application.

Sizing

s-8vcpu-16gb. This is not the same shape as the other analytics packages: 8 GiB wedged the machine hard enough that sshd could not fork, and upstream’s own hobby minimum is 16 GiB.

Reproducibility

The application and plugin server images are pinned to one commit. They share a Postgres schema, and floating tags on either side let the node process query columns the application’s migrations never created.

Service	Image	Role
`web`	`posthog/posthog` @ `82ea668`	Django application, served by NGINX Unit
`worker`	same commit	Celery worker and scheduler
`plugins`	`posthog/posthog-node` @ `82ea668`	Consumes the ingestion topic into ClickHouse
`capture`	`ghcr.io/posthog/posthog/capture`	Rust event capture; writes to Kafka
`kafka`	`redpandadata/redpanda:v25.1.9`	Event bus, as upstream uses
`clickhouse`	`clickhouse-server:26.6.2.158`	Columnar store, with embedded Keeper
`temporal`	`temporalio/auto-setup:1.26.2`	Workflow engine, persisted in the same Postgres
`db`	`postgres:17-alpine`	Relational store for PostHog and Temporal
`redis`	`redis:7.2-alpine`	Celery broker, configured `noeviction`
`caddy`	`caddy:2.11.4`	TLS termination and ingestion routing

Redis is the broker, not a cache: upstream’s hobby compose caps it at 200 MB with allkeys-lru, which silently discards queued ingestion tasks under pressure. Here it refuses to evict. ClickHouse runs Keeper embedded rather than as a separate ZooKeeper, which is what makes ReplicatedMergeTree possible on one node.

Desired state and credentials

colors.yml is the only file you edit, and it holds non-secret values only: the profile, DigitalOcean region, size and SSH sources, the host, every image tag, the data directories, and the backup destination, schedule and retention. See the configuration reference for every key.

Variable	Purpose
`COLORS_PAR_DO_TOKEN`	DigitalOcean compute and firewall
`COLORS_PAR_CLOUDFLARE_API_TOKEN`	DNS record for the configured host
`COLORS_PAR_POSTHOG_SECRET_KEY`	Django signing key; signs sessions and password-reset tokens
`COLORS_PAR_POSTHOG_POSTGRES_PASSWORD`	PostgreSQL password, percent-encoded into `DATABASE_URL`
`COLORS_PAR_POSTHOG_OIDC_RSA_PRIVATE_KEY`	RSA key for OAuth setup; the application exits at startup without it
`COLORS_PAR_POSTHOG_ENCRYPTION_SALT_KEYS`	Encrypts stored fields; shared by the application and plugin server
`COLORS_PAR_POSTHOG_BACKUP_R2_ACCESS_KEY_ID` `COLORS_PAR_POSTHOG_BACKUP_R2_SECRET_ACCESS_KEY`	R2 credentials for the backup bucket

Generate the last two with openssl genrsa 2048 and openssl rand -hex 16. Every one of these is required for a real create: the run fails before contacting a provider rather than starting a stack that serves a UI and silently ingests nothing.

Never export COLORS_PAR_PROFILE. The profile keys remote state.
Never edit or commit generated .colors/ output.
Keep compute-prevent-destroy: true in committed desired state.

Migrations and the schema checkpoint

Applying PostHog’s Django migrations from zero takes over an hour, CPU-bound and single-threaded. The package therefore commits checkpoint.sql: the schema plus the django_migrations rows, in plain SQL, stamped with the commit it was taken from.

It is restored only into a database with no tables, and only when its stamped commit matches the commit the application image reports. The ordinary migrate then runs regardless, so a checkpoint behind the image applies the delta and heals forward. It is never faked. A checkpoint from a divergent commit is skipped rather than planting migrations the image has never heard of, because PostHog’s migrate command stops on orphaned migrations and prompts for input no converge can answer.

Migrations run with the application stopped: the image’s entrypoint migrates on startup, so bringing the web container up first puts two runners on one schema.

Acceptance and operations

./green create
ssh <droplet> 'cd /opt/posthog && docker compose ps'
ssh <droplet> 'cd /opt/posthog && docker compose logs --tail=50 plugins'
ssh <droplet> 'systemctl status posthog-backup.timer'

Acceptance runs at the end of create and reports only what it checked:

HTTPS with certificate verification. /_health/ must answer over a valid certificate; the check never disables verification.
Capture is read back from ClickHouse. A synthetic event is posted with the project’s API token and the events table must grow. A 2xx with no stored row is reported as dropped — the case that hides a broken ingestion pipeline behind a healthy-looking API. With no project provisioned the verdict is not-configured, which passes and claims nothing.
The backup drill is confirmed in R2. The unit is started and the profile prefix must then contain a non-empty object newer than the run.

Expect a first converge to take a while: the migration set is large, and NGINX Unit boots four Django workers before /_health/ answers. The health gate allows twenty minutes for that.

Backups, restoration, and disaster recovery

A systemd timer runs nightly at 02:30 UTC. Each run takes a compressed pg_dump and a native ClickHouse BACKUP into the server’s allowed path on the bind mount, then uploads both to R2 under the profile prefix. ClickHouse is never captured with a hot tar: that races the server’s merges and yields an archive that cannot be restored.

To recover, restore the PostgreSQL dump into an empty PostgreSQL 17 database and the ClickHouse archive with RESTORE, converge a replacement at the same pinned commit, and verify both the UI and event ingestion before moving DNS. Never test restoration over live data.

Prevent-destroy is not a backup. Deleting compute does not authorize deleting R2 objects, the discovered default VPC, unrelated DNS, or any other provider resource. Every image is pinned: the application and plugin server to one commit, capture by digest, and the datastores to explicit versions.

Development and source

bb test
bb golden
bb golden:accept
./scripts/launcher.sh

The suite asserts the properties that were expensive to learn: that migrations precede the health gate, that the ClickHouse reload is keyed on what the server has actually loaded rather than on this run’s change flags, that the application and plugin server tags are equal, and that the checkpoint is bound to its commit. Source on GitHub. Read every golden diff — never accept generated output merely to make the suite pass.