Skip to content
Deedboxlatest

Run on Kubernetes

This guide shows you how to run Deedbox with several replicas on Kubernetes.

Alert on the health check; do not probe with it

Section titled “Alert on the health check; do not probe with it”
builder.Services.AddHealthChecks().AddDeedboxHealthChecks();

The check is unhealthy when a projection or subscription is stalled, or when events wait and its checkpoint has not moved for StallAfter (10 minutes by default). A projection that is behind but moving, or rebuilding, is healthy.

A restart does not fix a stall, and neither does taking a pod out of the Service. A subscription stalls when the service it calls is down. With the check on a liveness probe, Kubernetes then restarts every pod in a loop until that service is back. With the check on a readiness probe, every pod leaves the Service and your API is down. So keep the check off both probes, and give it its own path:

// The probes: the process answers. They leave the Deedbox check out.
app.MapHealthChecks("/healthz", new() { Predicate = check => check.Name != "deedbox" });
// For alerts: unhealthy while a projection or subscription is stalled or stuck.
app.MapHealthChecks("/health/deedbox", new() { Predicate = check => check.Name == "deedbox" });
livenessProbe:
httpGet: { path: /healthz, port: 8080 }
readinessProbe:
httpGet: { path: /healthz, port: 8080 }

Point your monitoring at /health/deedbox and alert when it is unhealthy. The stalled projection and poison event runbooks tell you what to do then.

Replicas need no extra setup. Each batch of a projection runs while its replica holds the projection’s checkpoint row, so the projections spread across replicas and each runs on one replica at a time.

To keep background work off your web pods, turn the runner off there and run a worker deployment with it on:

builder.Services.AddDeedbox(es => es
.UsePostgres(connStr)
.Stream<Cart>(s => s.Events<ItemAdded, CheckedOut>())
.Runner(r =>
{
r.BatchSize = 500;
r.MaxPollDelay = TimeSpan.FromSeconds(5);
r.HandlerRetries = 5;
r.StallAfter = TimeSpan.FromMinutes(10);
}));

Set r.Enabled = false in the web pods.

Choose one:

  • ApplySchemaOnStartup(): every pod applies pending migrations under a database lock, so concurrent pods apply them once.
  • A Kubernetes Job before the rollout: deedbox schema apply --provider postgres --connection "$DEEDBOX_CONNECTION".

Without either, a pod fails at start-up with the exact fix (DBX001). A newer schema than the build is fine, so old pods keep running during a rolling deploy.

env:
- name: DEEDBOX_MASTER_KEY
valueFrom:
secretKeyRef: { name: deedbox, key: master-key }

Back the Secret up outside the cluster. Losing every copy of the master key loses all personal data.