Skip to main content

Command Palette

Search for a command to run...

Stop Letting "YOLO Deployments" Break Your Cluster: Hello, Kyverno!

The "Why are we doing this?" talk.

Updated
5 min read
Stop Letting "YOLO Deployments" Break Your Cluster: Hello, Kyverno!
M

DevOps Engineer by day, YAML debugger by night. I help turn “it works on my machine” into “it works in production” by automating infrastructure, building CI/CD pipelines, and keeping cloud systems happy and scalable. I enjoy breaking things safely (in staging), fixing them properly (in prod), and writing about real-world DevOps lessons—what worked, what didn’t, and what I wish I knew earlier. If it involves Docker, Kubernetes, or reducing pager alerts, I’m probably interested.

Introduction: The Silent Guardian of the API Server

Maintaining a Kubernetes cluster often feels like a constant battle against configuration drift. As teams scale, the anxiety of "who deployed what and why" grows. Without a gatekeeper, your API server is essentially an open door to any configuration, regardless of how insecure or inefficient it might be.

This is where Admission Controllers step in. Think of them as the middleware of the Kubernetes world—an essential layer of logic that scrutinizes every request after it has passed authentication and authorization, but before the state is persisted to the etcd database. This specific timing makes them the ultimate arbiter of what actually enters your cluster’s "source of truth."

An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object, but only after the request is authenticated and authorized. In simpler terms, admission controllers can be thought of as middleware that can validate, mutate, or reject requests to the Kubernetes API.

The "Wild West" of K8s: Why we can't have nice things

At the scale of an large company, Kubernetes can quickly become a "Wild West." Without strict policy enforcement, clusters are prone to systemic risks. Some researches also identifies misconfigurations—specifically overly permissive RBAC and unbounded service accounts—as the leading cause of security breaches.

Beyond security, there is the silent financial leak of "Resource Asymmetry." The kube-scheduler relies on a complex scoring algorithm to assign pods to nodes based on CPU and memory. However, because this scoring is highly variable, developers often "play it safe" by over-estimating resource requests to avoid performance issues. This leads to a scenario where "spending outstripping use" becomes the norm.

The danger of "Missing Requests and Limits" is particularly acute; without these guardrails, a single resource-hungry pod can trigger a denial-of-service for its neighbors. This evolving threat landscape, combined with the risk of insider threats, makes automated gatekeeping the architect's path to sanity.

Kyverno vs. OPA: Why I refuse to learn another language (Rego)

When choosing a policy engine, the industry generally gravitates toward two heavyweights: Kyverno and OPA Gatekeeper. However, the friction of learning a specialized query language like Rego (used by OPA) often slows down security adoption for YAML-native teams.

The tech team at Adevinta recently highlighted a "functional failure" in OPA Gatekeeper’s mutation capabilities that led them to migrate. Specifically, Gatekeeper's Assign feature failed because it could not modify fields based on contextual data—information residing outside the specific field being observed. While OPA requires a complex external data provider setup for this, Kyverno handles it natively within the same manifest.

It’s just YAML! (The moment you fall in love)

The Kyverno advantage is simple: it feels like a natural extension of the Kubernetes experience. Because policies are written in native YAML, they integrate seamlessly with kubectl, Helm, and GitOps workflows. It is a more intuitive way to handle governance than traditional programming-heavy engines.

We are also witnessing a major shift with the arrival of CEL (Common Expression Language). In Kubernetes v1.33, ValidatingAdmissionPolicy has officially reached V1 (GA). This is a game-changer because it allows for declarative validation directly in the vanilla API server without external HTTP webhooks. By removing the need for these callouts, architects can eliminate network latency and webhook failure modes entirely. Even OPA Gatekeeper is acknowledging this shift by adding CEL support, signaling a move toward standardized, high-performance logic.

3-Minute Install: From "Zero" to "Governance"

Getting started is faster than your coffee cools. You can go from an unmanaged cluster to a governed environment using the standard Helm workflow for Kyverno:

  1. Add the Repo: helm repo add kyverno https://kyverno.github.io/kyverno/

  2. Update: helm repo update

  3. Install: helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace

The Lifecycle of a Request

To master admission control, you must understand the technical sequence of a request:

  1. Mutating Phase: The controller modifies the request (e.g., injecting sidecars or team labels).

  2. Schema Validation: The API server performs a structural JSON check to ensure the resource is well-formed.

  3. Validating Phase: The controller checks the request against security rules and finally accepts or rejects it.

💡
Pro Tip: Always start in Audit Mode. This allows you to assess the impact of policies without breaking existing developer workflows. Once you have sanitized the environment, flip the switch to Enforce Mode.

Your First Policy: Ban the default namespace

Isolating workloads is the first step toward maturity. Kubernetes ships with a static admission controller called NamespaceLifecycle that acts as a basic safeguard. It prevents the accidental deletion of three critical system-reserved namespaces: default, kube-system, and kube-public.

However, true governance requires moving beyond defaults to enforce ResourceQuotas and custom labels. This prevents one team from monopolizing node resources. From there, follow the experimental insights from S&P 500 deployments to harden your securityContext:

  • Enforce Rootless Mode: Prohibit deployments where runAsNonRoot is false.

  • Identity Mapping: Explicitly define runAsUser and runAsGroup (e.g., set to 1000) to ensure workloads never run under a root identity.

  • Immutable Filesystems: Enforce readOnlyRootFilesystem: true to prevent attackers from persisting malicious scripts or exfiltrating data.

Conclusion: The Future of Sovereign Clusters

The evolution of Kubernetes is moving toward Zero Trust and "Sovereign Clusters"—environments where an enterprise maintains absolute sovereignty over its standards across multi-cluster platforms. By implementing these gatekeeping rules today, you lay the foundation for AI-driven threat detection and automated compliance.

If your API server disappeared tomorrow, would your policy engine know how to rebuild the trust?

💡
If you liked this article follow for more, and wait for the rest of the series, I will write more about each policy type and more…