GREEN · RED · BLUE PACKAGE SKILL · VULTR · NEON · CADDY

n8n Package Skill

A reproducible Green, Red, and Blue workflow for self-hosted n8n on one Vultr instance — behind Caddy TLS, with Code nodes isolated in an external task runner, and its database a colocated self-hosted Neon whose layers and WAL live in Cloudflare R2.

The database's durable copy is object storage. n8n is normally run on SQLite — fine until two workflows finish at once — or on a colocated Postgres whose durability is the instance's disk. Here the storage tier is Neon: the pageserver uploads layers and the safekeeper offloads WAL to <profile>/data/ in R2, and the tenant and timeline identities live in colors.yml, so a rebuilt host re-attaches the same identities.

The interesting claim, and the measurement

Storage/compute separation is usually assumed too slow for an OLTP write pattern, and n8n writes an execution row per run. Measured on a vhp-8c-16gb-amd with a declared workload mix — API-shaped calls, Code nodes at 8 MB to trigger n8n's payload duplication, and binary payloads through the filesystem path:

Measure	Result	Gate
executions in 5 minutes	7950, 0 failed	≥ 500
SQL round-trip p95 / p99	75 ms / 80 ms	150 / 500
host memory / disk	12% / 9%	≤ 85% / 80%

Reproduced across two independent runs with identical percentiles. The p95→p99 gap of 5 ms is the point: the split adds no visible tail latency at this rate. Honest scope: five minutes, one host, ten concurrent workflows. It does not establish behaviour over days, or during a pageserver restart under load.

Quick start

npx skills add getcolors/n8n
cp .agents/skills/package-n8n-green/green ./green
chmod +x green
./green build
./green create --dry-run

Three implementations of one model — Clojure/Babashka, TypeScript/Bun, Python/uv — take the same verbs and the same colors.yml. Swap package-n8n-green/green for package-n8n-red/red or package-n8n-blue/blue; scripts/parity.sh renders every fixture through all three and diffs the trees byte for byte, so "the same" is a checked claim rather than an intention.

Build renders reproducible output and dry-run walks the workflow without provider side effects. Real creation and deletion require explicit authorization.

Architecture

Compute

One Vultr instance and a firewall opening 22, 80 and 443 — and 80/443 only to Cloudflare's published ranges. In keygen mode the account SSH key resource is named after the profile and lives in this deployment's state.

Application tier

n8n 2.36.9 behind Caddy, with an external task runner so a Code node executes in its own container rather than in the main process. Every image pinned by tag and digest, and the runner's version must equal n8n's.

Storage tier

The storage broker, the pageserver, one safekeeper, and a Postgres 17 compute node under compute_ctl — all bound to loopback, all writing to R2 under the deployment's own prefix.

Acceptance

Server-side: the tenant attached, fresh WAL in R2, a workflow read back out of Neon, a Code node executed on the runner, the origin certificate, sshd's negative controls. Workstation-side: the SSH tunnel through the generated alias.

No second copy of the storage tier

The Neon tier is not reimplemented here. This package SHA-pins getcolors/neon and renders its Ansible templates straight out of it — compose.yml, main.yml, pageserver.toml, the compute spec, the bootstrap and rotation scripts — with no file copied into this repository. Each colour reaches the same templates its own way: green off the classpath, red out of the installed package's red/resources, blue out of package_neon_blue/resources.

Two mechanics make it work. n8n's own services arrive as a Compose override installed beside the upstream compose.yml, so every unchanged upstream command operates on the one merged project without any -f flags. And the imported play targets hosts: neon while this package's targets hosts: n8n, with the single host in both inventory groups.

The trade is real coupling: bumping the neon pin can break this package. That is deliberate — the golden renders the merged tree, so a pin bump shows up as a reviewable diff instead of a surprise on a host.

Exposure

Port	Exposure	Purpose
22	`vultr-ssh-sources`	Key-only SSH: convergence, recovery, and the client tunnel
80 / 443	`vultr-http-sources`, by default Cloudflare's ranges alone	Caddy: the editor, the REST API, and every webhook endpoint
55433 / 3080	loopback	Postgres and the compute API, reached through the tunnel
9898 / 6400 / 7676 / 5454 / 50051	loopback	Pageserver, safekeeper, and the storage broker

Caddy is the only service in the merged project that publishes beyond loopback, and it aborts any request whose Host it does not serve — so a scan of the origin address gets a closed connection rather than a login page. Its trusted_proxies list is only trustworthy because the firewall admits nothing else on 80/443: trusting X-Forwarded-For from an open origin is trusting the header's author.

Restricting the origin to Cloudflare and leaving the record unproxied are mutually exclusive. Caddy answers the ACME HTTP-01 challenge on :80, and with the record unproxied that challenge arrives from Let's Encrypt's own addresses, which the firewall drops. The converge still succeeds and the first HTTPS request finds a certificate that was never issued — so validation refuses the combination up front.

Three buckets, three credentials

The package refuses to converge when one R2 credential would reach OpenTofu state, live Neon data and backups alike. Measured on a live host, that pair could list, write and delete in the state bucket and delete backup sets — and a backup a compromised host can erase is not a backup. Mint bucket-scoped tokens for COLORS_PAR_NEON_R2_* and COLORS_PAR_N8N_BACKUP_R2_*; COLORS_PAR_R2_* then has one job and never leaves your workstation. A first converge that predates those tokens can set r2-credential-sharing: shared-accepted — but only as a deliberate line in committed desired state, visible in a diff.

The database role password, the n8n owner password and the task-runner token are generated on the server and are not operator credentials; read them over SSH from /etc/neon/secrets/ and /etc/n8n/secrets/.

The encryption key outlives the host. N8N_ENCRYPTION_KEY encrypts every credential n8n stores. n8n writes it into /home/node/.n8n/config on first boot and refuses to start ever after if the environment disagrees — so one bad first boot poisons the data directory, and the error appears on the next boot rather than the one that caused it. Keep a copy somewhere that is not this machine before the first converge.

The machine keypair and the SSH config block

The deployment owns its SSH key, following the workspace SSH Keypair Standard. Leave vultr-ssh-keys out of colors.yml and the first real create generates ~/.ssh/<profile>, registers it at Vultr under the profile name, and a successful delete removes it last — never before. Supplying vultr-ssh-keys opts out entirely.

Convergence also writes one ~/.ssh/config block, following the workspace SSH Config Standard, so ssh <profile> reaches the host with no address, user or -i flag — and the acceptance step rides exactly that alias, so the block is proven, not assumed. Delete removes the block before destroying the instance, the reverse of the keypair: a block that outlives its host is stale but harmless, while a key removed early locks you out of a machine that still exists.

Drills

Acceptance runs on every converge and fails it. These do not:

ssh <profile> /opt/neon/n8n-smoke.sh          # the acceptance gates on demand
ssh <profile> /opt/neon/n8n-soak.sh           # load, against the declared thresholds
ssh <profile> /opt/neon/n8n-rehearsal.sh      # restore into an isolated scratch stack
ssh <profile> /opt/neon/n8n-prune-drill.sh    # execution retention, isolated
ssh <profile> /opt/neon/n8n-restart-drill.sh recreate

Recovery, honestly

Neon uploads WAL to R2 continuously, but a rebuilt safekeeper does not recover its offloaded WAL — the walproposer bootstraps it from the compute basebackup. A destroyed host therefore falls back to the logical backup set, and the backup interval is the real RPO: six hours by default, not nightly.

Failure	Recovers from	RPO
pageserver local state lost	R2 layers + safekeeper replay	~0
compute lost	recreate from pageserver	~0
whole host lost	the backup set in R2	one backup interval

n8n-rehearsal.sh proves the whole path: restore both artifacts into an isolated scratch stack, boot the pinned image, log in, and execute a workflow whose node carries a stored credential — because a credential's value cannot be read back to test decryption, only used.

Safety

Credentials are COLORS_PAR_* variables in the gitignored .envrc.private — never in colors.yml.
Never export COLORS_PAR_PROFILE; the profile keys remote state.
.colors/ is generated output: never edit or commit it.
compute-prevent-destroy: true guards deletion behind a one-run override — and delete leaves the R2 data in place.

Source on GitHub · Configuration reference