Back to Blog
DevOps

Kubernetes Production Deployment Strategies: A War Story

Cap
14 min read
kubernetesdeploymentproductioncanaryblue-green

Kubernetes Production Deployment Strategies: A War Story

How we went from 2-hour maintenance windows to zero-downtime deployments serving 10M+ users

🔥 The Problem That Started It All

March 15th, 2023 - 3:47 AM

$ kubectl get pods -n production
NAME                           READY   STATUS    RESTARTS   AGE
api-server-7d8f9c6b4d-xkj2m   0/1     Error     0          2m
api-server-7d8f9c6b4d-9m4n7   0/1     Error     0          2m
api-server-7d8f9c6b4d-5k8p3   0/1     Error     0          2m

# 💀 All pods failing after deployment
# 🚨 10 million users can't access the platform
# ⏰ Revenue loss: $50,000 per minute

This was our wake-up call. Our naive deployment strategy—replace all pods at once—had just taken down our entire production system. Here's how we fixed it and built bulletproof deployment strategies.

📊 Current State: Battle-Tested Metrics

After 18 months of iteration, here's what we achieved:

MetricBeforeAfterImprovement
Deployment Success Rate67%99.7%+48.8%
Mean Time to Recovery45 minutes2.3 minutes-94.9%
Zero-Downtime Deployments0%100%∞
Rollback Time8 minutes15 seconds-96.9%
User-Facing Errors During Deploy15.2%0.01%-99.9%

🎯 Strategy 1: Rolling Updates (The Foundation)

Rolling updates became our baseline—safe, predictable, but not perfect for critical services.

Configuration

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 12
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%     # Never remove more than 3 pods
      maxSurge: 25%          # Never add more than 3 extra pods
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
        version: "v1.2.3"
    spec:
      containers:
      - name: api-server
        image: myregistry/api-server:v1.2.3
        ports:
        - containerPort: 8080
        
        # 🚀 Critical: Proper health checks
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
          
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 2
          
        # 🎯 Resource limits prevent noisy neighbors
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
            
        # 🔧 Graceful shutdown
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]
              
      # 🚀 Pod disruption budget
      terminationGracePeriodSeconds: 30
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-server-pdb
spec:
  minAvailable: 75%  # Always keep 9/12 pods running
  selector:
    matchLabels:
      app: api-server

Health Check Implementation

// health.go - Comprehensive health checking
package main

import (
    "context"
    "database/sql"
    "encoding/json"
    "net/http"
    "time"
)

type HealthChecker struct {
    db    *sql.DB
    redis *redis.Client
    deps  []Dependency
}

type HealthStatus struct {
    Status      string            `json:"status"`
    Timestamp   time.Time         `json:"timestamp"`
    Version     string            `json:"version"`
    Dependencies map[string]bool   `json:"dependencies"`
    Uptime      string            `json:"uptime"`
}

// Liveness probe - "Is the app running?"
func (h *HealthChecker) LivenessHandler(w http.ResponseWriter, r *http.Request) {
    status := HealthStatus{
        Status:    "ok",
        Timestamp: time.Now(),
        Version:   AppVersion,
        Uptime:    time.Since(StartTime).String(),
    }
    
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(status)
}

// Readiness probe - "Can the app handle traffic?"
func (h *HealthChecker) ReadinessHandler(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
    defer cancel()
    
    deps := make(map[string]bool)
    allHealthy := true
    
    // Check database
    if err := h.db.PingContext(ctx); err != nil {
        deps["database"] = false
        allHealthy = false
    } else {
        deps["database"] = true
    }
    
    // Check Redis
    if _, err := h.redis.Ping(ctx).Result(); err != nil {
        deps["redis"] = false
        allHealthy = false
    } else {
        deps["redis"] = true
    }
    
    // Check external dependencies
    for _, dep := range h.deps {
        if !dep.IsHealthy(ctx) {
            deps[dep.Name] = false
            allHealthy = false
        } else {
            deps[dep.Name] = true
        }
    }
    
    status := HealthStatus{
        Status:       func() string { if allHealthy { return "ready" } else { return "not_ready" } }(),
        Timestamp:    time.Now(),
        Version:      AppVersion,
        Dependencies: deps,
        Uptime:       time.Since(StartTime).String(),
    }
    
    if allHealthy {
        w.WriteHeader(http.StatusOK)
    } else {
        w.WriteHeader(http.StatusServiceUnavailable)
    }
    
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(status)
}

Rolling Update Results:

  • ✅ Zero-downtime: 98.5% success rate
  • ⚠️ Still risky for critical breaking changes
  • 📊 Average deployment time: 3.2 minutes

🔷 Strategy 2: Blue-Green Deployments (The Safe Bet)

For critical services where we needed instant rollback capability.

Implementation with Argo Rollouts

# blue-green-rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payment-service
spec:
  replicas: 10
  strategy:
    blueGreen:
      # Service routing
      activeService: payment-service-active
      previewService: payment-service-preview
      
      # Auto-promotion (optional)
      autoPromotionEnabled: false
      
      # Promotion after successful tests
      prePromotionAnalysis:
        templates:
        - templateName: success-rate
        args:
        - name: service-name
          value: payment-service-preview
          
      # Keep blue env for quick rollback
      scaleDownDelaySeconds: 600
      
      # Time to manually verify before promotion
      promotionPolicy:
        timeoutSeconds: 300
        
  selector:
    matchLabels:
      app: payment-service
      
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      containers:
      - name: payment-service
        image: myregistry/payment-service:latest
        ports:
        - containerPort: 8080
        env:
        - name: DB_CONNECTION_POOL_SIZE
          value: "20"
        - name: REDIS_MAX_CONNECTIONS
          value: "100"
        
        # Enhanced health checks for financial service
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 45
          periodSeconds: 15
          
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 2  # Require 2 consecutive successes
          
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
---
# Active service (receives production traffic)
apiVersion: v1
kind: Service
metadata:
  name: payment-service-active
spec:
  selector:
    app: payment-service
  ports:
  - port: 80
    targetPort: 8080
---
# Preview service (for testing green deployment)
apiVersion: v1
kind: Service
metadata:
  name: payment-service-preview
spec:
  selector:
    app: payment-service
  ports:
  - port: 80
    targetPort: 8080

Analysis Templates for Automated Testing

# success-rate-analysis.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 30s
    count: 5
    successCondition: result[0] >= 0.95  # 95% success rate required
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        query: |
          sum(rate(http_requests_total{service="{{args.service-name}}",status=~"2.."}[5m])) /
          sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))
          
  - name: avg-response-time
    interval: 30s
    count: 5
    successCondition: result[0] <= 500  # Max 500ms response time
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        query: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket{service="{{args.service-name}}"}[5m])) by (le)
          ) * 1000

Deployment Script

#!/bin/bash
# deploy-blue-green.sh

set -euo pipefail

SERVICE_NAME="payment-service"
NEW_IMAGE="$1"
NAMESPACE="production"

echo "🚀 Starting Blue-Green deployment for $SERVICE_NAME"
echo "📦 New image: $NEW_IMAGE"

# Update rollout with new image
kubectl patch rollout $SERVICE_NAME -n $NAMESPACE -p \
  "{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"$SERVICE_NAME\",\"image\":\"$NEW_IMAGE\"}]}}}}"

echo "⏳ Waiting for rollout to start..."
kubectl rollout status rollout/$SERVICE_NAME -n $NAMESPACE --timeout=300s

# Monitor the preview service
echo "🔍 Running smoke tests on preview service..."
PREVIEW_URL="http://$(kubectl get svc ${SERVICE_NAME}-preview -n $NAMESPACE -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"

# Health check
curl -f "$PREVIEW_URL/health" || {
  echo "❌ Health check failed"
  kubectl argo rollouts abort $SERVICE_NAME -n $NAMESPACE
  exit 1
}

# Load test
echo "📊 Running load test..."
k6 run --vus 10 --duration 2m tests/load-test.js --env ENDPOINT="$PREVIEW_URL" || {
  echo "❌ Load test failed"
  kubectl argo rollouts abort $SERVICE_NAME -n $NAMESPACE
  exit 1
}

# Integration tests
echo "🧪 Running integration tests..."
pytest tests/integration/ --endpoint="$PREVIEW_URL" || {
  echo "❌ Integration tests failed"
  kubectl argo rollouts abort $SERVICE_NAME -n $NAMESPACE
  exit 1
}

echo "✅ All tests passed! Promoting to production..."
kubectl argo rollouts promote $SERVICE_NAME -n $NAMESPACE

echo "🎉 Blue-Green deployment completed successfully!"

Blue-Green Results:

  • ✅ Instant rollback capability
  • ✅ 99.9% deployment success rate
  • ⚠️ Requires 2x resources during deployment
  • 📊 Promotion confidence: 100%

🐦 Strategy 3: Canary Deployments (The Gradual Approach)

For user-facing applications where we needed to monitor real user impact.

Advanced Canary with Traffic Splitting

# canary-rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: frontend-app
spec:
  replicas: 20
  strategy:
    canary:
      # Traffic splitting configuration
      canaryService: frontend-app-canary
      stableService: frontend-app-stable
      
      # Gradual traffic increase
      steps:
      - setWeight: 5    # Start with 5% traffic
      - pause:
          duration: 300s  # 5 minutes
      - setWeight: 10
      - pause:
          duration: 300s
      - setWeight: 20
      - pause:
          duration: 600s  # 10 minutes for monitoring
      - setWeight: 50
      - pause:
          duration: 600s
      - setWeight: 100
      
      # Automatic analysis at each step
      analysis:
        templates:
        - templateName: error-rate
        - templateName: response-time
        - templateName: user-satisfaction
        
        # Start analysis after initial traffic
        startingStep: 2
        
        # Abort conditions
        args:
        - name: canary-hash
          valueFrom:
            podTemplateHashValue: Latest
            
      # Istio traffic management
      trafficRouting:
        istio:
          virtualService:
            name: frontend-app-vs
            routes:
            - primary
            
  selector:
    matchLabels:
      app: frontend-app
      
  template:
    metadata:
      labels:
        app: frontend-app
    spec:
      containers:
      - name: frontend-app
        image: myregistry/frontend-app:latest
        ports:
        - containerPort: 3000
        
        # Frontend-specific health checks
        livenessProbe:
          httpGet:
            path: /api/health
            port: 3000
          initialDelaySeconds: 30
          
        readinessProbe:
          httpGet:
            path: /api/ready
            port: 3000
          initialDelaySeconds: 10
          
        # Resource configuration for frontend
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "200m"
            
        # Environment variables
        env:
        - name: NODE_ENV
          value: "production"
        - name: API_BASE_URL
          value: "https://api.example.com"
        - name: FEATURE_FLAGS_ENDPOINT
          value: "https://flags.example.com"

Istio Virtual Service for Traffic Splitting

# istio-virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: frontend-app-vs
spec:
  hosts:
  - app.example.com
  http:
  - match:
    - headers:
        canary:
          exact: "true"
    route:
    - destination:
        host: frontend-app-canary
        port:
          number: 80
      weight: 100
      
  - route:
    - destination:
        host: frontend-app-stable
        port:
          number: 80
      weight: 100  # This will be modified by Argo Rollouts
    - destination:
        host: frontend-app-canary
        port:
          number: 80
      weight: 0    # This will be modified by Argo Rollouts

Advanced Canary Analysis

# canary-analysis.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-analysis
spec:
  args:
  - name: canary-hash
  
  metrics:
  # Error rate monitoring
  - name: error-rate
    interval: 60s
    count: 5
    successCondition: result[0] <= 0.01  # Max 1% error rate
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        query: |
          sum(rate(http_requests_total{
            app="frontend-app",
            rollouts_pod_template_hash="{{args.canary-hash}}",
            status=~"5.."
          }[5m])) /
          sum(rate(http_requests_total{
            app="frontend-app",
            rollouts_pod_template_hash="{{args.canary-hash}}"
          }[5m]))
          
  # Response time P95
  - name: response-time-p95
    interval: 60s
    count: 5
    successCondition: result[0] <= 1000  # Max 1s P95 response time
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        query: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket{
              app="frontend-app",
              rollouts_pod_template_hash="{{args.canary-hash}}"
            }[5m])) by (le)
          ) * 1000
          
  # Custom business metrics
  - name: conversion-rate
    interval: 120s
    count: 3
    successCondition: result[0] >= 0.05  # Min 5% conversion rate
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        query: |
          sum(rate(user_conversions_total{
            app="frontend-app",
            rollouts_pod_template_hash="{{args.canary-hash}}"
          }[10m])) /
          sum(rate(user_sessions_total{
            app="frontend-app",
            rollouts_pod_template_hash="{{args.canary-hash}}"
          }[10m]))
          
  # User satisfaction (from real user monitoring)
  - name: user-satisfaction
    interval: 300s
    count: 2
    successCondition: result[0] >= 7.0  # Min 7.0/10 satisfaction score
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        query: |
          avg(user_satisfaction_score{
            app="frontend-app",
            rollouts_pod_template_hash="{{args.canary-hash}}"
          })

Deployment Automation with Notifications

#!/bin/bash
# deploy-canary.sh

set -euo pipefail

SERVICE_NAME="frontend-app"
NEW_IMAGE="$1"
NAMESPACE="production"
SLACK_WEBHOOK="$SLACK_DEPLOYMENT_WEBHOOK"

function send_slack_notification() {
    local message="$1"
    local color="$2"
    
    curl -X POST -H 'Content-type: application/json' \
        --data "{
            \"attachments\": [{
                \"color\": \"$color\",
                \"text\": \"$message\",
                \"fields\": [{
                    \"title\": \"Service\",
                    \"value\": \"$SERVICE_NAME\",
                    \"short\": true
                }, {
                    \"title\": \"Image\",
                    \"value\": \"$NEW_IMAGE\",
                    \"short\": true
                }]
            }]
        }" \
        $SLACK_WEBHOOK
}

function abort_deployment() {
    echo "❌ Aborting canary deployment"
    kubectl argo rollouts abort $SERVICE_NAME -n $NAMESPACE
    send_slack_notification "🚨 Canary deployment ABORTED for $SERVICE_NAME" "danger"
    exit 1
}

# Set up cleanup trap
trap abort_deployment ERR

echo "🚀 Starting Canary deployment for $SERVICE_NAME"
send_slack_notification "🚀 Starting canary deployment for $SERVICE_NAME" "warning"

# Update rollout
kubectl patch rollout $SERVICE_NAME -n $NAMESPACE -p \
  "{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"$SERVICE_NAME\",\"image\":\"$NEW_IMAGE\"}]}}}}"

# Wait for first canary step (5% traffic)
echo "⏳ Waiting for initial canary deployment..."
kubectl argo rollouts get rollout $SERVICE_NAME -n $NAMESPACE --watch

# Monitor key metrics during canary
while true; do
    PHASE=$(kubectl get rollout $SERVICE_NAME -n $NAMESPACE -o jsonpath='{.status.phase}')
    
    if [[ "$PHASE" == "Succeeded" ]]; then
        echo "✅ Canary deployment completed successfully!"
        send_slack_notification "✅ Canary deployment SUCCEEDED for $SERVICE_NAME" "good"
        break
    elif [[ "$PHASE" == "Degraded" ]] || [[ "$PHASE" == "Failed" ]]; then
        echo "❌ Canary deployment failed or degraded"
        send_slack_notification "❌ Canary deployment FAILED for $SERVICE_NAME" "danger"
        exit 1
    elif [[ "$PHASE" == "Paused" ]]; then
        CURRENT_STEP=$(kubectl get rollout $SERVICE_NAME -n $NAMESPACE -o jsonpath='{.status.currentStepIndex}')
        WEIGHT=$(kubectl get rollout $SERVICE_NAME -n $NAMESPACE -o jsonpath='{.status.canaryStatus.stableRS}' | jq '.weight // 0')
        
        echo "⏸️  Canary paused at step $CURRENT_STEP (${WEIGHT}% traffic)"
        
        # Check if manual intervention is needed
        if [[ $CURRENT_STEP -ge 4 ]]; then  # 50% traffic step
            echo "🤔 Manual review required for 50% traffic promotion"
            send_slack_notification "🤔 Manual review needed: Canary at 50% traffic for $SERVICE_NAME" "warning"
            
            read -p "Continue with full promotion? (y/N): " -n 1 -r
            echo
            if [[ $REPLY =~ ^[Yy]$ ]]; then
                kubectl argo rollouts promote $SERVICE_NAME -n $NAMESPACE
            else
                abort_deployment
            fi
        fi
    fi
    
    sleep 30
done

echo "🎉 Canary deployment completed!"

Canary Results:

  • ✅ Minimal blast radius (5% initial exposure)
  • ✅ Real user feedback integration
  • ✅ 99.8% deployment success rate
  • 📊 Average rollback time: 15 seconds

🛡️ Strategy 4: Feature Flags + Deployments

The ultimate safety net—deploy code without activating features.

Feature Flag Service Integration

// feature_flags.go
package main

import (
    "context"
    "time"
    "github.com/launchdarkly/go-server-sdk/v6"
)

type FeatureFlagService struct {
    client *ldapi.LDClient
}

func NewFeatureFlagService(sdkKey string) (*FeatureFlagService, error) {
    client, err := ldapi.MakeClient(sdkKey, 5*time.Second)
    if err != nil {
        return nil, err
    }
    
    return &FeatureFlagService{client: client}, nil
}

func (f *FeatureFlagService) IsEnabled(flagKey string, userContext map[string]interface{}) bool {
    user := ldapi.User{
        Key: userContext["user_id"].(string),
        Custom: userContext,
    }
    
    return f.client.BoolVariation(flagKey, user, false)
}

// Gradual rollout based on user percentage
func (f *FeatureFlagService) GetRolloutPercentage(flagKey string) int {
    user := ldapi.User{Key: "system"}
    return f.client.IntVariation(flagKey, user, 0)
}

// Usage in HTTP handler
func (h *Handler) PaymentHandler(w http.ResponseWriter, r *http.Request) {
    userID := getUserID(r)
    userContext := map[string]interface{}{
        "user_id": userID,
        "country": getUserCountry(r),
        "plan":    getUserPlan(userID),
    }
    
    // Check if new payment processing is enabled
    if h.flags.IsEnabled("new-payment-processor", userContext) {
        h.processPaymentV2(w, r)
    } else {
        h.processPaymentV1(w, r)
    }
}

Deployment with Feature Flag Coordination

# feature-flag-deployment.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: feature-flag-update
spec:
  template:
    spec:
      containers:
      - name: flag-updater
        image: myregistry/flag-updater:latest
        env:
        - name: LAUNCHDARKLY_SDK_KEY
          valueFrom:
            secretKeyRef:
              name: feature-flags
              key: sdk-key
        command:
        - /bin/sh
        - -c
        - |
          # Wait for new pods to be ready
          kubectl wait --for=condition=ready pod -l app=payment-service --timeout=300s
          
          # Gradually enable feature flag
          for percentage in 5 10 25 50 100; do
            echo "Setting new-payment-processor to ${percentage}%"
            curl -X PATCH \
              -H "Authorization: api-key $LAUNCHDARKLY_API_KEY" \
              -H "Content-Type: application/json" \
              -d "{\"percentage\": $percentage}" \
              "https://app.launchdarkly.com/api/v2/flags/production/new-payment-processor"
            
            # Monitor for 5 minutes
            sleep 300
            
            # Check error rates
            ERROR_RATE=$(prometheus-query "error_rate{feature='new-payment-processor'}")
            if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
              echo "Error rate too high, rolling back feature flag"
              curl -X PATCH \
                -H "Authorization: api-key $LAUNCHDARKLY_API_KEY" \
                -H "Content-Type: application/json" \
                -d '{"percentage": 0}' \
                "https://app.launchdarkly.com/api/v2/flags/production/new-payment-processor"
              exit 1
            fi
          done
          
      restartPolicy: Never

🚨 Emergency Procedures

Instant Rollback Playbook

#!/bin/bash
# emergency-rollback.sh

SERVICE_NAME="$1"
NAMESPACE="production"

echo "🚨 EMERGENCY ROLLBACK for $SERVICE_NAME"

# Check current rollout status
ROLLOUT_TYPE=$(kubectl get rollout $SERVICE_NAME -n $NAMESPACE -o jsonpath='{.spec.strategy}' | jq -r 'keys[0]')

case $ROLLOUT_TYPE in
  "blueGreen")
    echo "🔄 Performing Blue-Green rollback"
    kubectl argo rollouts abort $SERVICE_NAME -n $NAMESPACE
    ;;
    
  "canary")
    echo "🔄 Performing Canary rollback"
    kubectl argo rollouts abort $SERVICE_NAME -n $NAMESPACE
    ;;
    
  *)
    echo "🔄 Performing standard rollback"
    kubectl rollout undo deployment/$SERVICE_NAME -n $NAMESPACE
    ;;
esac

# Wait for rollback to complete
kubectl rollout status deployment/$SERVICE_NAME -n $NAMESPACE --timeout=120s

# Disable feature flags
echo "🚫 Disabling all experimental feature flags"
curl -X POST "$FEATURE_FLAG_DISABLE_ALL_ENDPOINT" \
  -H "Authorization: Bearer $FEATURE_FLAG_API_KEY"

# Send alerts
echo "📢 Sending rollback notifications"
curl -X POST $SLACK_WEBHOOK -d '{
  "text": "🚨 EMERGENCY ROLLBACK completed for '$SERVICE_NAME'",
  "attachments": [{
    "color": "danger",
    "fields": [{
      "title": "Service",
      "value": "'$SERVICE_NAME'",
      "short": true
    }, {
      "title": "Time",
      "value": "'$(date)'",
      "short": true
    }]
  }]
}'

echo "✅ Emergency rollback completed"

📊 Monitoring and Observability

Custom Metrics Dashboard

# deployment-dashboard.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: deployment-dashboard
data:
  dashboard.json: |
    {
      "dashboard": {
        "title": "Deployment Health",
        "panels": [
          {
            "title": "Deployment Success Rate",
            "type": "stat",
            "targets": [{
              "expr": "sum(rate(deployment_status_total{status=\"success\"}[1h])) / sum(rate(deployment_status_total[1h]))"
            }]
          },
          {
            "title": "Rollback Frequency",
            "type": "graph",
            "targets": [{
              "expr": "sum(rate(deployment_rollback_total[1h])) by (service)"
            }]
          },
          {
            "title": "Deployment Duration",
            "type": "heatmap",
            "targets": [{
              "expr": "histogram_quantile(0.95, sum(rate(deployment_duration_seconds_bucket[5m])) by (le))"
            }]
          }
        ]
      }
    }

🎯 Lessons Learned

✅ What Works

  1. Start Simple: Rolling updates for non-critical services
  2. Progressive Enhancement: Blue-Green for critical services, Canary for user-facing
  3. Feature Flags: The ultimate safety net for risky changes
  4. Automated Testing: Never deploy without comprehensive analysis
  5. Monitoring: Real-time metrics are non-negotiable

❌ What Doesn't Work

  1. Big Bang Deployments: All-or-nothing approaches fail
  2. Manual Testing Only: Human testing doesn't scale
  3. Ignoring Health Checks: Proper probes are critical
  4. No Rollback Plan: Always have an escape route
  5. Skipping Resource Limits: Noisy neighbors kill deployments

📈 Impact on Business

  • User Experience: 99.99% uptime during deployments
  • Developer Productivity: 15 deployments per day (vs 1 per week)
  • Revenue Protection: Zero downtime = $0 revenue loss
  • Time to Market: Features reach users 10x faster
  • Confidence: Team deploys fearlessly

🚀 Next Steps

Advanced Patterns We're Exploring

  1. Progressive Delivery: GitOps + Canary + Feature Flags
  2. Multi-Cluster Deployments: Cross-region rollouts
  3. A/B Testing Integration: Deployment-driven experiments
  4. Chaos Engineering: Automated failure injection during deployments

The Bottom Line: Great deployment strategies aren't just about technology—they're about confidence. When you can deploy fearlessly, you ship faster, break less, and deliver more value to users.

What deployment challenges are you facing? Share your war stories in the comments!

WY

Cap

Senior Golang Backend & Web3 Developer with 10+ years of experience building scalable systems and blockchain solutions.

View Full Profile →
Back to Blog