BLUE PACKAGE SKILL · VULTR · GITHUB WAREHOUSE

GitHub DWH Package Skill

A reproducible Blue workflow for a single-host GitHub organization warehouse: dlt extracts every repository a credential can see, dbt builds ClickHouse marts, Lightdash serves the dashboards, and a PocketBase control plane schedules and records whole runs.

Strict ownership boundaries. PocketBase stores schedules and whole-run history only — one run record is one ./blue run. Blue owns workflow routing, systemd owns process supervision, journald owns full logs, and Git owns desired state.

Quick start

npx skills add getcolors/github-dwh
cp .agents/skills/package-github-dwh-blue/blue ./blue
chmod +x blue
./blue build
./blue create --dry-run

Build renders reproducible output and dry-run walks the workflow without provider side effects. Real creation, loads, and deletion require explicit authorization.

Architecture

Infrastructure

OpenTofu creates one Ubuntu 24.04 Vultr instance and a firewall exposing 22, 80, and 443, plus Cloudflare DNS for the control plane and the analytics host. State lives in a shared Cloudflare R2 bucket keyed by profile.

Warehouse

dlt loads repositories, commits, organization events, Actions runs, and Package Skill manifests into ClickHouse; dbt builds and tests the staging and marts models on every run.

Analytics

Lightdash serves the semantic layer, charts, and repository dashboard from read-only ClickHouse marts, with PostgreSQL metadata and chart artifacts in a dedicated R2 bucket.

Control plane

PocketBase provides authenticated schedules, run-now, and run history behind Caddy TLS; a fixed systemd timer dispatches complete Blue workflow invocations and a read-only gateway serves redacted journald logs.

Desired state and lifecycle

Non-secret configuration belongs only in colors.yml: the GitHub organization and resources, both hostnames, ClickHouse databases, the Lightdash R2 bucket, the dispatch calendar, and the Vultr and state-backend boundary.

./blue build
./blue create --dry-run
./blue create
./blue run
./blue delete

Supply credentials only as matching COLORS_PAR_* environment variables.
Never export COLORS_PAR_PROFILE or edit generated .colors/.
Keep compute-prevent-destroy: true; deletion remains separately guarded.
run performs a real GitHub load and dbt build unless --dry-run is present.

Operations

Operators start recurring or manual loads from PocketBase; the host launcher executes each as a transient systemd unit, and the control plane links Lightdash and every log source.

ssh root@SERVER 'cd /opt/github-dwh && docker compose ps'
ssh root@SERVER 'journalctl -u github-dwh-dispatch.service --since -1h'
curl https://ANALYTICS_HOST/api/v1/health

One run is one workflow. Do not add task DAGs, per-step queues, sensors, or retry policy to PocketBase — retry a failure by creating a new complete run. GitHub organization events are a bounded feed; recover history from ClickHouse backups, not the API.

Development and source

uv run pytest
./scripts/golden.sh
./scripts/launcher.sh

Source on GitHub. Review every golden diff before accepting it.