Run on Kubernetes
This guide shows you how to run Deedbox with several replicas on Kubernetes.
Alert on the health check; do not probe with it
Section titled “Alert on the health check; do not probe with it”builder.Services.AddHealthChecks().AddDeedboxHealthChecks();The check is unhealthy when a projection or subscription is stalled, or when events wait and its checkpoint has not moved for StallAfter (10 minutes by default). A projection that is behind but moving, or rebuilding, is healthy.
A restart does not fix a stall, and neither does taking a pod out of the Service. A subscription stalls when the service it calls is down. With the check on a liveness probe, Kubernetes then restarts every pod in a loop until that service is back. With the check on a readiness probe, every pod leaves the Service and your API is down. So keep the check off both probes, and give it its own path:
// The probes: the process answers. They leave the Deedbox check out.app.MapHealthChecks("/healthz", new() { Predicate = check => check.Name != "deedbox" });
// For alerts: unhealthy while a projection or subscription is stalled or stuck.app.MapHealthChecks("/health/deedbox", new() { Predicate = check => check.Name == "deedbox" });livenessProbe: httpGet: { path: /healthz, port: 8080 }readinessProbe: httpGet: { path: /healthz, port: 8080 }Point your monitoring at /health/deedbox and alert when it is unhealthy. The stalled projection and poison event runbooks tell you what to do then.
Run several replicas
Section titled “Run several replicas”Replicas need no extra setup. Each batch of a projection runs while its replica holds the projection’s checkpoint row, so the projections spread across replicas and each runs on one replica at a time.
To keep background work off your web pods, turn the runner off there and run a worker deployment with it on:
builder.Services.AddDeedbox(es => es .UsePostgres(connStr) .Stream<Cart>(s => s.Events<ItemAdded, CheckedOut>()) .Runner(r => { r.BatchSize = 500; r.MaxPollDelay = TimeSpan.FromSeconds(5); r.HandlerRetries = 5; r.StallAfter = TimeSpan.FromMinutes(10); }));Set r.Enabled = false in the web pods.
Change the schema
Section titled “Change the schema”Choose one:
ApplySchemaOnStartup(): every pod applies pending migrations under a database lock, so concurrent pods apply them once.- A Kubernetes Job before the rollout:
deedbox schema apply --provider postgres --connection "$DEEDBOX_CONNECTION".
Without either, a pod fails at start-up with the exact fix (DBX001). A newer schema than the build is fine, so old pods keep running during a rolling deploy.
Keep the master key in a Secret
Section titled “Keep the master key in a Secret”env: - name: DEEDBOX_MASTER_KEY valueFrom: secretKeyRef: { name: deedbox, key: master-key }Back the Secret up outside the cluster. Losing every copy of the master key loses all personal data.