Chaos Engineering: Breaking Things on Purpose (Before They Break You)

We started running chaos experiments after a client's payment service went down for 45 minutes because a single Redis node ran out of memory. The service was "designed for high availability" with replicas, health checks, and auto-scaling. None of it worked as expected when the actual failure happened. That's the gap chaos engineering closes — the distance between what you think your system does during failure and what it actually does.

What Chaos Engineering Actually Is (Not Just "Kill Random Stuff")

Chaos engineering is disciplined experimentation on distributed systems. The key word is disciplined. You're not randomly pulling plugs. You're forming a hypothesis about how your system handles a specific failure, running a controlled experiment, and observing whether reality matches your expectations.

The scientific method, applied to infrastructure:

1. Define steady state — What does "healthy" look like? (e.g., p99 latency under 200ms, error rate below 0.1%)
2. Hypothesize — "If Redis primary fails, the replica promotes within 10 seconds and the app continues serving with no errors"
3. Inject failure — Kill the Redis primary
4. Observe — Did the failover happen? How long did it take? What happened to in-flight requests?
5. Learn — Either your hypothesis was correct (great, you have evidence) or it wasn't (great, you found a bug before your customers did)

Both outcomes are wins. That's the mindset shift that makes chaos engineering work.

The Steady-State Hypothesis: Getting It Right

The steady-state hypothesis is where most teams mess up. "The system should be fine" is not a hypothesis. You need measurable, observable criteria.

Good vs. Bad Hypotheses

Types of Chaos Experiments

Start with the simple ones and work your way up. Seriously — we've seen teams jump straight to multi-AZ failure and learn nothing because they couldn't even handle a single pod restart.

Level 1: Infrastructure Failures

• Pod/container kill — Does the deployment recover? How long until the replacement pod is healthy?
• Node drain/termination — Do pods reschedule? Does the Kubernetes scheduler spread them correctly?
• Disk fill — Does the app handle full disk gracefully? Do alerts fire?
• CPU/memory stress — Do resource limits protect other pods on the node? Does auto-scaling kick in?

Level 2: Network Failures

• Network latency injection — Add 500ms latency to database calls. Do timeouts and circuit breakers work?
• Packet loss — 5% packet loss on inter-service calls. How do retries behave? Do they cascade?
• DNS failure — Corrupt DNS resolution. Does the app cache DNS appropriately? Or does one DNS blip cascade to every service?
• Network partition — Split the network between services. Does the system detect the split? How does it recover when the partition heals?

Level 3: Application Failures

• Dependency outage — Kill a downstream service entirely. Does the caller degrade gracefully or cascade-fail?
• Slow dependency — Make a dependency respond in 10 seconds instead of 100ms. This is worse than a hard failure because the caller's thread pool fills up waiting
• Data corruption — Return malformed responses from a dependency. Does the caller validate input or blindly trust it?

Level 4: Multi-Component Failures

• AZ failure simulation — Drain all nodes in one availability zone. Does the system stay up with reduced capacity?
• Cascading failure — Kill a service that triggers retry storms in its callers. Do circuit breakers prevent the cascade?
• Clock skew — Advance the clock on some nodes. Do JWT tokens expire unexpectedly? Do cron jobs fire twice?

Tools: What We've Used and What We Recommend

Running Game Days: The Practical Guide

A game day is a scheduled chaos experiment session where the team intentionally breaks things and practices incident response. Think of it as a fire drill for your infrastructure.

Before the Game Day

1. Pick 3-5 experiments — Ranked by impact and likelihood. Don't try to test everything
2. Define blast radius controls — What's the maximum impact you're willing to accept? At what point do you abort?
3. Ensure observability — Can you actually see what's happening? Dashboards prepped, alerts tuned, logs accessible
4. Notify stakeholders — Customer support, PMs, anyone who might see impact. "We're running chaos experiments from 2-4 PM, you might see brief latency spikes"
5. Prepare rollback — Know exactly how to stop the experiment instantly

During the Game Day

• One person runs the experiment, one person monitors dashboards, one person takes notes
• Start with the mildest experiment. Build confidence before escalating
• If anything unexpected happens, stop immediately. You've already learned something valuable
• Time-box each experiment: 15-minute max

After the Game Day

• Write up findings within 24 hours (memory fades fast)
• For each experiment: hypothesis, actual result, gap analysis, action items
• Prioritize fixes and assign owners. Chaos without follow-up is just breaking things
• Share results broadly. Game day findings often reveal issues relevant to multiple teams

Chaos in Kubernetes: A Practical Example

Here's a real Litmus chaos experiment we run for clients' K8s deployments.

# litmus-pod-delete.yaml
# Experiment: Kill a random pod from the user-service deployment
# Hypothesis: HPA scales a replacement within 30s, p99 stays under 500ms
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: user-service-pod-delete
  namespace: production
spec:
  appinfo:
    appns: production
    applabel: app=user-service
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '30'      # Kill pods for 30 seconds
            - name: CHAOS_INTERVAL
              value: '10'      # Every 10 seconds
            - name: FORCE
              value: 'false'   # Graceful termination (test shutdown hooks)
            - name: PODS_AFFECTED_PERC
              value: '30'      # Kill 30% of pods (1 of 3)
        probe:
          - name: check-latency
            type: promProbe
            mode: Continuous
            promProbe/inputs:
              endpoint: http://prometheus:9090
              query: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="user-service"}[1m]))
              comparator:
                type: float
                criteria: '<='
                value: '0.5'   # p99 must stay under 500ms

The probe is the key part — it automatically checks whether p99 latency stayed within bounds during the experiment. If it breached 500ms, the experiment is marked as failed and you know you have a resilience gap to fix.

Chaos Maturity Model

Where is your organization on this spectrum? Be honest — most are at Level 1 or 2.

What We Learned Breaking Our Own Systems

We run chaos experiments on our own cloud infrastructure and client environments. Some surprising findings:

• Circuit breakers often aren't configured correctly. We found that 40% of the circuit breakers we tested either had thresholds too high (triggered after the system was already cascading) or reset too quickly (closed before the dependency actually recovered)
• Health checks lie. A pod can pass its health check while being completely unable to serve traffic — because the health check endpoint doesn't exercise the same code path as real requests. Test health checks as part of your chaos experiments
• Retry storms are the #1 cause of cascading failures. Service A fails, services B through G all retry simultaneously, overwhelming service A's recovery. Exponential backoff with jitter isn't optional — it's a requirement
• DNS failures cause the widest blast radius. When we simulated DNS issues, more services broke than with any other failure type. And most teams had zero DNS resilience

Frequently Asked Questions