GREEN · RED · BLUE PACKAGE SKILL · AUTOMQ

AutoMQ Package Skill

A reproducible workflow for a three-node AutoMQ cluster using colors-compute, the Kafka 3.9 wire protocol over brokers whose storage tier is object storage rather than replicated disks. Shipped in Clojure, TypeScript and Python, rendering the same bytes from one colors.yml.

Replication factor 1 is the design, not a mistake. AutoMQ acknowledges a produce once the record is in object storage, so replicas would add cost and write amplification without adding durability. Upstream ships RF=1 for the same reason. The three nodes buy a controller quorum, partition failover, and throughput. What RF=1 does not buy is availability, which is why acceptance kills a broker that leads a specific partition and measures how long that partition stays unwritable.

Quick start

npx skills add getcolors/automq
cp .agents/skills/package-automq-green/green ./green
chmod +x green
./green build
./green create --dry-run

Build renders reproducible output and dry-run walks the workflow without provider side effects. Real creation and deletion require explicit authorization.

Substitute package-automq-red/red or package-automq-blue/blue for the same verbs on Bun or uv. The three colours render byte-identical artifacts, and scripts/parity.sh is what proves it on every change.

Architecture

Compute

Three machines on a private network provisioned by colors-compute, with public SSH and client access and private quorum rules. The controller quorum and inter-broker replication never touch the public interface.

Storage tier

Two application buckets, each owned by one cluster. The package adopts empty R2 buckets or creates private AWS S3, GCS or OCI buckets with scoped credentials. State occupies a separate bucket.

Identity

Four principals, no anonymous access anywhere. The controller listener uses PLAIN from a static JAAS entry; inter-broker and public listeners use SCRAM bootstrapped into the metadata log at genesis.

Acceptance

Host and public-client gates verify records, authentication and authorization. Targeted failover abruptly kills the selected partition leader and measures recovery from before the fault.

OCI deployment

Set automq-storage-provider: oci and automq-storage-managed: true to create private data and ops buckets with their own OCI user, group, policy, customer secret key and API signing key. Use a third OCI bucket for state with provider-backend: oci and oci-bucket-mode: managed. OpenTofu uses OCI's S3 compatibility endpoint for state.

Bucket ownership has the same deployment lifetime as the SSH keypair. After broker cleanup, guarded delete checks the application buckets against their recorded state, removes objects and incomplete multipart uploads, then deletes the buckets and their IAM resources. The state bucket is last.

OCI lease updates use native signed requests because the S3 compatibility endpoint ignores PUT If-Match. Before genesis, a gate verifies competing creates and stale native ETag rejection. It waits up to 15 minutes for new credentials to propagate.

On September 12, 2026, three ARM A1.Flex brokers (1 OCPU and 8 GB each, across three Frankfurt availability domains) completed three full converges: two on 5aac5a3 and one on final source 6b76c01. All three passed 16 public gates; the final run also captured all nine host gates and unchanged firewall application on every host. Abrupt leader kills recovered partition writes in 11, 11 and 10 seconds; records and committed offsets survived. IP-SAN TLS, scoped state-access denials and native lease checks passed. Reboot verification preserved the repaired platform firewall and application rules.

The offsets topic leader was not forced onto the failed broker. Full running-cluster cleanup has not been repeated; September 11 capacity failures and partial-deployment cleanup remain separate historical evidence.

The immutable automq-oci evidence carries the desired state and deployment evidence. The OCI configuration reference lists credentials, endpoints and ownership checks.

Connecting

The bootstrap name carries one A record per node; each broker advertises its own name, and one Let's Encrypt certificate covers all of them.

kcat -b <automq-host>:9092 \
  -X security.protocol=SASL_SSL \
  -X sasl.mechanism=SCRAM-SHA-512 \
  -X sasl.username=automq \
  -X sasl.password=<from automq-credential> -L

The password is generated on the server and printed by automq-credential, a root-only command deliberately separate from automq-status. Routine status checks keep credentials out of scrollback.

Authentication is not authorization

Port 9092 faces the internet, so SCRAM alone is not the boundary. The cluster runs StandardAuthorizer with allow.everyone.if.no.acl.found=false, and the public client principal holds prefixed Describe/Read/Write on topics and Describe/Read on groups. It cannot Create, Alter or perform ClusterAction. Acceptance asserts the denials, not only the grants.

One certificate issuer

Node 0 alone runs ACME and is the only host that receives a Cloudflare token, publishing the result to the ops bucket for the others to pull. Three nodes issuing independently would race on the shared _acme-challenge record for the bootstrap name and delete one another's proof. They would also put a zone-editing credential on every broker. Restarts are ordered by a conditional-create lease in object storage, because a node checking its own quorum health cannot order actors it cannot see.

Exposure

Port	Exposure	Purpose
22	`automq-ssh-sources`	Key-only SSH: convergence and recovery
9092	`automq-kafka-sources`	The public Kafka endpoint: SASL_SSL, SCRAM-SHA-512, ACL-gated
9093	VPC	KRaft controller quorum
9094	VPC	Inter-broker traffic

delete never empties the buckets. They hold the cluster’s data, and a teardown that silently erased them would make an accidental delete unrecoverable. Purging is a separate, deliberate action.

Source

github.com/getcolors/automq · reference deployment automq-vultr · getcolors.ai