Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler adds and removes replicas to keep an observed metric near a target. More Pods, not bigger ones, that is the horizontal part, and it is what makes it suitable for stateless services.

It is one of the more genuinely useful controllers in Kubernetes, and also one people wire up once and never verify.

The formula

Everything the HPA does follows from one line:

Three replicas averaging 90% CPU against a 50% target gives ceil(3 × 1.8) = 6 replicas. The controller re-evaluates every 15 seconds by default.

Two consequences worth noticing. It is proportional, so a large overshoot produces a large jump rather than one replica at a time. And it depends on currentReplicas, which is why an HPA cannot scale up from zero. Anything times zero is zero. Scale-to-zero needs KEDA.

It requires resource requests

This is the single most common reason an HPA does nothing.

CPU and memory targets are expressed as a percentage of the request. A container with no CPU request has no denominator, so utilisation is undefined and the HPA reports <unknown> and refuses to act.

kubectl get hpa

<unknown> means one of exactly two things: metrics-server is not installed, or the target Pods have no resource requests.

kubectl top pods                # fails entirely if metrics-server is missing
kubectl get deployment payments-api -o jsonpath='{.spec.template.spec.containers[0].resources}'

A working HPA

Use autoscaling/v2. The older v1 supports CPU only.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payments-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

kubectl apply -f hpa.yaml
kubectl get hpa payments-api -w

Remove replicas from the Deployment manifest once an HPA manages it. Otherwise every kubectl apply resets the replica count to whatever the file says, and the HPA scales it back up moments later. A loop that looks like flapping and is really two controllers disagreeing. If you use GitOps, exclude the field from diffing.

Controlling how it scales

By default the HPA scales up quickly and down slowly. A deliberate asymmetry, since being briefly over-provisioned is much cheaper than being under-provisioned. The behavior block makes it explicit:

spec:
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0      # react immediately
      policies:
        - type: Percent
          value: 100                     # at most double
          periodSeconds: 15
        - type: Pods
          value: 4                       # or at most +4
          periodSeconds: 15
      selectPolicy: Max                  # whichever allows more
    scaleDown:
      stabilizationWindowSeconds: 300    # wait 5 minutes of calm
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

stabilizationWindowSeconds on scale-down is the setting that matters most. It makes the HPA use the highest recommendation from the last N seconds, which prevents a brief dip in traffic from removing capacity that is needed again a minute later. The 300-second default is sensible; shortening it causes thrashing.

Beyond CPU

CPU is a poor proxy for load in plenty of services. An I/O-bound API may be saturated at 20% CPU. autoscaling/v2 supports better signals.

Memory, though be careful: many runtimes hold memory rather than returning it, so memory rarely falls after a spike and the HPA will not scale back down.

    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70

Custom metrics from your application, via Prometheus Adapter. Requests per second is usually far more meaningful than CPU:

    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"

External metrics such as queue depth, which is the correct signal for a worker pool:

    - type: External
      external:
        metric:
          name: sqs_messages_visible
          selector:
            matchLabels:
              queue: payments-jobs
        target:
          type: AverageValue
          averageValue: "30"

With several metrics listed, the HPA computes a desired replica count for each and takes the highest. That is the safe behaviour. Any one signal indicating pressure is enough to scale up, and every signal must be calm to scale down.

Nodes have to exist

An HPA that wants 20 replicas on a cluster with room for 12 leaves eight Pods Pending. Pod autoscaling and node autoscaling are separate concerns, and you generally need both:

• Cluster Autoscaler grows and shrinks node groups.
• Karpenter provisions right-sized instances directly, and responds faster.

Set maxReplicas to a number your cluster or your budget can actually absorb. It is a safety limit as much as a scaling parameter.

Diagnosing

kubectl describe hpa payments-api

Those three conditions answer most questions directly:

• ScalingActive: False: metrics unavailable. metrics-server missing, or no resource requests.
• ScalingLimited: True: it wants to scale further but minReplicas/maxReplicas is in the way.
• AbleToScale: False: it cannot modify the target, often a conflicting controller or a missing scale subresource.

kubectl get events --field-selector involvedObject.name=payments-api

What it will not do

• Scale from zero. The formula multiplies by current replicas. Use KEDA.
• Scale StatefulSets safely by default. It works mechanically, but adding a database replica is rarely just a replica count.
• Work alongside VPA on the same metric. VPA resizing Pods while HPA counts them on CPU causes both to fight. Use VPA for memory and HPA for CPU, or keep them apart.

KEDA is worth knowing about: it extends the HPA with scale-to-zero and dozens of event sources. Kafka lag, SQS depth, RabbitMQ, cron. For queue-driven workers it is usually the better tool.

Quick reference

Key takeaways

• desiredReplicas = ceil(current × (currentMetric / target)): proportional, and unable to start from zero.
• Resource requests are mandatory. Utilisation is a percentage of the request; with no request it is <unknown>.
• <unknown> means metrics-server is missing or requests are unset. Only those two.
• Remove replicas from a Deployment managed by an HPA, or the two controllers fight on every apply.
• stabilizationWindowSeconds on scale-down prevents thrashing. The 300s default is sensible.
• With several metrics, the highest recommendation wins.
• Pod autoscaling needs node autoscaling or the extra replicas sit Pending.

Check your understanding

<MCQExercise exercise={{ id: 'k8s-hpa-unknown', title: 'Targets showing unknown', description: 'kubectl get hpa shows TARGETS as <unknown>/50% and the replica count never changes. kubectl top pods works correctly. What is wrong?', options: [ { id: 'a', text: 'The target Pods have no CPU resource requests, so utilisation as a percentage of request cannot be computed' }, { id: 'b', text: 'metrics-server is not installed in the cluster' }, { id: 'c', text: 'The HPA is using autoscaling/v1, which cannot report CPU targets' }, { id: 'd', text: 'minReplicas is set higher than the current replica count' }, ], correctId: 'a', explanation: 'A Utilization target is expressed as a percentage of the container's CPU request, so with no request there is no denominator and the value is undefined. The detail that rules out the other candidate is that kubectl top works, that proves metrics-server is healthy and serving data, leaving missing requests as the only explanation.', wrongExplanations: { b: 'Missing metrics-server would make kubectl top fail outright with an error about the metrics API. Since it works, metrics are being collected fine.', c: 'autoscaling/v1 supports CPU targets; it is the versions beyond CPU that it lacks. Either version would report a real number if requests were set.', d: 'A minReplicas above the current count causes an immediate scale up to that floor, not an unknown metric reading.', }, }} />

<MCQExercise exercise={{ id: 'k8s-hpa-replicas-conflict', title: 'Replica count that keeps resetting', description: 'An HPA scales a Deployment to 8 replicas. After every CI deploy the count drops to 3 and then climbs back. The Deployment manifest specifies replicas: 3. What is happening?', options: [ { id: 'a', text: 'Each apply resets the replica field to 3, and the HPA then scales back up. Two controllers writing the same field' }, { id: 'b', text: 'The HPA resets to minReplicas after any change to the Deployment' }, { id: 'c', text: 'Rolling updates always restart autoscaling from the manifest value' }, { id: 'd', text: 'The HPA cannot manage a Deployment that has a replicas field and is being ignored entirely' }, ], correctId: 'a', explanation: 'The replicas field has two writers. kubectl apply sets it to the value in the file, and the HPA sets it to its own computed value moments later, so the count oscillates on every deploy and briefly drops capacity. Remove replicas from the manifest once an HPA owns the Deployment, and configure GitOps tooling to ignore that field.', wrongExplanations: { b: 'The HPA recomputes from current metrics on its normal interval and has no special reset behaviour tied to Deployment changes.', c: 'A rolling update replaces Pods but does not alter the desired replica count. The change here comes from the apply, not the rollout.', d: 'The HPA works perfectly well, that is why the count climbs back. The problem is that its work is undone on each apply.', }, }} />