GitOps Fundamentals: Why Your CI/CD Pipeline Is Lying to You
I built a complete GitOps platform from scratch—two EKS clusters, ArgoCD with the App-of-Apps pattern, Terraform modules, Karpenter autoscaling, Kyverno policies, and 352 automated tests. The whole thing deploys in under an hour and syncs changes to production in five minutes.
But the technology isn't the story. The story is what happened before I built it—years of 3 AM incidents, configuration drift nobody could explain, and the slow realization that our CI/CD pipelines were fundamentally broken. Not broken in the sense that they didn't work. Broken in the sense that they gave us confidence we didn't deserve.
This article is for anyone who has ever asked: "Wait, what's actually running in production right now?" and couldn't get a straight answer.
The Problem with Push-Based CI/CD
Here's how most deployment pipelines work:
- Developer pushes code to Git
- CI system builds and tests
- CI system pushes artifacts to production
- Everyone hopes for the best
I call this "push and pray." Jenkins, GitHub Actions, GitLab CI—they all do the same thing. They push changes to your infrastructure and then walk away. The pipeline succeeds, the notification fires, and everyone assumes the deployment worked.
But did it?
I once spent four hours debugging an incident caused by a developer who ran kubectl apply directly against production "just to test something real quick." The change never went through our pipeline. It wasn't in Git. The only evidence was buried in Kubernetes audit logs.
This is configuration drift—when what's running in production doesn't match what's defined in your source code. Push-based CI/CD creates drift because it treats deployment as an event rather than a state. The pipeline has no idea what happens after it runs.
What GitOps Actually Means
GitOps is a term coined in 2017, but the principles are older. At its core, GitOps is about treating Git as the single source of truth for your entire system—not just application code, but infrastructure, configuration, and policies.
The GitOps model has four core principles:
1. Declarative
Your entire system must be described declaratively. Instead of scripts that do things ("create a deployment"), you have manifests that describe things ("there should be a deployment").
2. Versioned and Immutable
Everything goes into Git. Every change gets committed, reviewed, and merged through the same workflows your application code uses. This gives you a complete audit trail for free.
3. Pulled Automatically
Here's where GitOps diverges from traditional CI/CD. Instead of pipelines pushing changes, agents running inside your clusters pull changes from Git. This improves security as your CI server doesn't need admin access to production clusters.
4. Continuously Reconciled
The agents continuously compare the actual state of your cluster against the desired state in Git. If there's any difference, they reconcile it. This is the key to eliminating drift.
Push vs. Pull: A Mental Model
Think of push-based CI/CD like a contractor who shows up, does some work, and leaves. You hope they did the job correctly, but you won't really know until you inspect it yourself.
GitOps is like having a caretaker who lives on the property. They know exactly how everything should look because they have the blueprints. They constantly walk around checking that things match the blueprints. If something's wrong, they fix it.
The Business Case for GitOps
In industries where compliance is mandatory, GitOps makes compliance almost automatic. Every change goes through a pull request and creates a permanent record.
Even outside regulated industries, GitOps delivers value:
- Faster recovery: Rolling back is just
git revert. - Reduced cognitive load: Developers don't need access to production clusters.
- Environment consistency: Staging and production are generated from the same source.
The Bottom Line
Your CI/CD pipeline isn't lying to you maliciously. It's just not designed to tell you the truth about what's actually running in production. It deploys and walks away.
GitOps changes the model. Git becomes the single source of truth. Agents continuously reconcile actual state against desired state. Drift becomes impossible.
Whether you use AWS, Docker, or manage complex DevOps workflows, GitOps provides the reliability you need.