Zero-Downtime Kubernetes Deployments for Financial Services: What the Docs Don't Tell You

Kubernetes rolling deployments are often described as zero-downtime by default. This is not accurate. A rolling deployment prevents total unavailability — it doesn't automatically prevent individual requests from failing during the transition.

There are four specific gaps between a rolling deployment and true zero-downtime:

• ▸. The new pod receives traffic before it is ready to serve requests.
• ▸. The old pod is terminated before in-flight requests complete.
• ▸. The load balancer still routes to a pod that has been marked for deletion.
• ▸. The application starts before external dependencies (database connections, cache) are initialised.

Each of these gaps is closeable with specific configuration. Here's how we configured Kubernetes for Dashen Super App's financial services.

Kubernetes will not route traffic to a pod until its readiness probe returns success. But many teams configure readiness probes that succeed too early — before the application is actually ready to serve traffic.

For a Go service using gRPC health checks:

readinessProbe:
  grpc:
    port: 9090
    service: "grpc.health.v1.Health"
  initialDelaySeconds: 5
  periodSeconds: 3
  failureThreshold: 3

The probe hits the gRPC health endpoint — which the service only marks as SERVING after it has successfully connected to the database, Redis, and all upstream dependencies. If any dependency is unavailable, the pod stays unready and receives no traffic.

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5

The /health/ready endpoint should check all dependencies — not just return 200. A service that can accept HTTP connections but cannot reach its database is not ready.

When Kubernetes terminates a pod, it sends SIGTERM. Without graceful shutdown handling, the process exits immediately, dropping all in-flight requests.

In a Go service:

func main() {
    server := &http.Server{Addr: ":8080", Handler: router}

go func() { if err := server.ListenAndServe(); err != http.ErrServerClosed { log.Fatal(err) } }()

// Wait for termination signal quit := make(chan os.Signal, 1) signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT) <-quit

// Graceful shutdown: wait for in-flight requests (30s timeout) ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel()

Also configure terminationGracePeriodSeconds in the deployment spec to match your shutdown timeout:

spec:
  terminationGracePeriodSeconds: 60

There is a race condition between Kubernetes removing a pod from its endpoint list and the load balancer (or kube-proxy) propagating that change. During this window — typically 5-15 seconds — new requests can still be routed to a pod that is shutting down.

The fix is a preStop lifecycle hook that adds a sleep before the application begins shutdown:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]

The sleep gives the load balancer time to deregister the pod before SIGTERM is sent. Combined with graceful shutdown, this eliminates the race condition.

The default rolling update strategy replaces pods one at a time. For financial services where availability is critical, configure maxUnavailable: 0 to ensure capacity never drops below 100%:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

This keeps all existing pods running while spinning up new ones. Traffic only shifts to new pods after their readiness probes pass. The deployment uses more resources temporarily (100% + 1 pod worth), but guarantees no reduction in serving capacity during the rollout.

The complete zero-downtime configuration for a financial service combines all four fixes:

• ▸. Readiness probe that checks all dependencies before accepting traffic
• ▸. Graceful shutdown handling SIGTERM and waiting for in-flight requests
• ▸. preStop sleep hook to allow load balancer deregistration
• ▸. RollingUpdate with maxUnavailable: 0 to prevent capacity reduction

This configuration has been running in production for Dashen Super App's payment services since deployment. We have completed over 200 rolling deployments without a single dropped payment request — in a system where dropping a payment request is a financial incident.