Automating Pod Disruption Budgets with Kyverno

<div class="toc"> <ul> <li><a href="#the-problem-forgotten-pdb">The Problem: The "Forgotten" PDB</a></li> <li><a href="#the-solution-policy-as-code-with-kyverno">The Solution: Policy as Code with Kyverno</a></li> <li><a href="#the-intelligent-policy">The Intelligent Policy</a></li> <li><a href="#breaking-down-the-magic">Breaking Down the Magic</a></li> <li><a href="#deploying-with-terraform">Deploying with Terraform</a></li> <li><a href="#the-gotcha-permissions">The "Gotcha": Permissions</a></li> <li><a href="#what-are-aggregated-clusterroles">What are Aggregated ClusterRoles?</a></li> <li><a href="#conclusion">Conclusion</a></li> </ul> </div> <p>At Zencity.io, our platform serves as a critical bridge between local governments and their residents. Our services need to be "always-on."</p> <p>Running a microservices architecture on Kubernetes at scale brings specific challenges. One of our biggest wins for cluster efficiency has been adopting <a href="https://aws.amazon.com/karpenter/">Karpenter</a> for autoscaling (Preferably official AWS documentation). Karpenter is excellent at reducing costs by aggressively consolidating nodes — packing pods tightly onto fewer instances when possible.</p> <p>However, this efficiency introduced a risk. Karpenter keeps consolidating nodes. If a specific microservice had all its replicas scheduled on the same node (often because it lacked explicit pod anti-affinity rules) and that node was selected for consolidation, Karpenter would drain it. Without a safety net, the drain operation would evict all replicas simultaneously, causing immediate downtime for that service.</p> <p>That safety net is the Pod Disruption Budget (PDB). Simply put, a PDB allows application owners to define the minimum number of replicas that must be available (or the maximum that can be unavailable) during voluntary disruptions, such as node drains for maintenance or upgrades.</p> <p>Here is what a basic PDB looks like for an application that requires at least 2 replicas to be online:</p> <pre><code class="language-yaml"> apiVersion: policy.io/v1beta1 kind: PodDisruptionBudget metadata: name: example-pdb namespace: example-namespace spec: minAvailable: 2 selector: matchLabels: app: example-app </code></pre> <h2 id="the-problem-forgotten-pdb">The Problem: The "Forgotten" PDB</h2> <p>A PDB is a standard Kubernetes resource, but it requires manual definition. We faced a classic operational gap across our tens of microservices:</p> <ol> <li><strong>The Karpenter Factor:</strong> Karpenter's automated node consolidation turned "rare" maintenance events into frequent, automated actions.</li> <li><strong>Human Error:</strong> Developers would create a new microservice but forget to define a PodDisruptionBudget.</li> <li><strong>Toil:</strong> Auditing every service manually to ensure it had a PDB was inefficient.</li> </ol> <h2 id="the-solution-policy-as-code-with-kyverno">The Solution: Policy as Code with Kyverno</h2> <p>We chose <a href="https://kyverno.io/" target="_blank">Kyverno</a>, a policy engine designed specifically for Kubernetes. Unlike other tools that require learning a separate language (like Rego), Kyverno uses Kubernetes-native YAML patterns. Our goal was simple: If a Deployment or StatefulSet has multiple replicas (>= 2) but NO matching PDB, create one automatically.</p> <p><em>Note: Specific feature requirements such as Kubernetes v1.20+ and Kyverno v1.9+ might be needed for this to function properly (This part could not be verified as it was not explicitly mentioned in the source).</em></p> <h2 id="the-intelligent-policy">The Intelligent Policy</h2> <p>The real challenge wasn't just generating a PDB; it was ensuring we didn't create duplicate PDBs or interfere with services that already had custom configurations. In our configuration, we set <code>maxUnavailable</code> to 20%, ensuring that during a disruption, the vast majority of replicas remain online.</p> <pre><code class="language-yaml"> apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: auto-generate-pdb spec: rules: - name: generate-pdb-for-deployments match: any: - resources: kinds: - Deployment - StatefulSet context: - name: existing_pdb_count apiCall: urlPath: "/apis/policy/v1/namespaces/{{request.namespace}}/poddisruptionbudgets" jmesPath: "items[?spec.selector.matchLabels == `{{request.object.spec.selector.matchLabels}}`]" preconditions: any: - key: "{{request.object.spec.replicas}}" operator: GreaterThanOrEquals value: 2 - key: "{{existing_pdb_count | length(@)}}" operator: Equals value: 0 generate: apiVersion: policy/v1 kind: PodDisruptionBudget name: "{{request.object.metadata.name}}-pdb" namespace: "{{request.namespace}}" synchronize: true data: spec: maxUnavailable: 20% selector: matchLabels: "{{request.object.spec.selector.matchLabels}}" </code></pre> <h3 id="breaking-down-the-magic">Breaking Down the Magic</h3> <p>The most powerful part of this policy is the <code>context</code> block. PDBs map by labels, not just by Deployment names. We use Kyverno's <code>apiCall</code> to query existing PDBs in the namespace, apply a label filter, and if the count is 0, we proceed to generate the resource. We also use <code>generateExisting: true</code>, which scans our cluster and backfills PDBs for older services.</p> <h2 id="deploying-with-terraform">Deploying with Terraform</h2> <p>At Zencity, we manage our infrastructure as code. We use <a href="https://www.devopsn.cloud/tech/terraform-consultancy">Terraform</a> to template and apply this policy across our environments. This allows us to keep our configuration dynamic.</p> <pre><code class="language-hcl"> resource "kubernetes_manifest" "kyverno_cluster_policy" { manifest = { "apiVersion" = "kyverno.io/v1" "kind" = "ClusterPolicy" "metadata" = { "name" = "auto-generate-pdb" } "spec" = { # Policy details... } } } </code></pre> <h2 id="the-gotcha-permissions">The "Gotcha": Permissions</h2> <p>Kyverno has a background controller that scans existing resources and generates new ones. Out of the box, it might not have permissions to manage PodDisruptionBudgets. The controller needs full management access (create, delete, update) to PDBs, not just read access.</p> <h3 id="what-are-aggregated-clusterroles">What are Aggregated ClusterRoles?</h3> <p>In Kubernetes, ClusterRole Aggregation is a modular way to extend permissions. Instead of modifying a massive, monolithic ClusterRole, you create smaller, specific ClusterRoles and label them. The Kubernetes control plane automatically aggregates (merges) these rules.</p> <pre><code class="language-yaml"> apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kyverno-pdb-manager labels: app.kubernetes.io/component: background-controller app.kubernetes.io/instance: kyverno app.kubernetes.io/part-of: kyverno spec: rules: - apiGroups: ["policy"] resources: ["poddisruptionbudgets"] verbs: ["create", "update", "delete", "get", "list", "watch"] </code></pre> <h2 id="conclusion">Conclusion</h2> <p>By implementing this policy, we moved from a "hope-based" reliability strategy to an automated one. Thanks to <a href="https://www.devopsn.cloud/tech/kubernetes-consultancy">Kubernetes</a> and Kyverno, we no longer worry if a developer forgot a PDB snippet or if a lack of pod anti-affinity will cause downtime. By applying similar <a href="https://www.devopsn.cloud/tech/aws-consultancy">AWS</a> and cloud automation practices alongside <a href="https://www.devopsn.cloud/">DevOpsN</a> principles, you can securely increase resource efficiency.</p> <p><i>Kaynak / Source: http://freedium-mirror.cfd/https://medium.com/zencity-engineering/automating-pod-disruption-budgets-with-kyverno-0a6bee7bbcca</i></p>