Ah, GitOps. Supposed to be the holy grail of DevOps, right?
Just commit, push, and watch the magic happen. Your infrastructure gets updated like clockwork. Everything versioned, everything automated. What could possibly go wrong?
Plenty.
Turns out, when you let automation take the wheel without enough checks, it’ll happily drive you off a cliff. We learned this the hard way—twice. Here’s what happened.
⚠️ Incident 1: The Great Deletion Debacle
Friday evening. Everyone’s wrapping up. One last merge before heading out—supposed to hit a test environment.
But it didn’t.
Instead, the change went straight to production. And with it? A Terraform script that wiped our live database.
No warning. No prompts. Just… gone.
Our monitoring stack (Prometheus and Grafana) stayed quiet. The app crashed. Other systems relying on that database followed. Within minutes, we were staring at a full-on outage that lasted four hours.
Getting things back? Painful. Our backups were buried in messy Terraform code left behind by a previous team. Felt more like digital archaeology than DevOps:
# Restoring from backup via Terraform
terraform import aws_db_instance.my_db arn:aws:rds:us-west-2:123456789012:db:mydatabase
terraform apply -var="environment=production"
What went wrong?
We had no guardrails. No environment checks. No role-based access controls. Just one bad merge—and the system did exactly what it was told.
What we fixed:
- Tagged all Terraform modules by environment (test, staging, prod).
- Added drift detection (
tfsec,terraform plan) to PRs. - Required senior reviewers for any production merges.
🔄 Incident 2: Dependency Hell
A month later, different mess. Our "Order Service" got upgraded to v2. Clean GitOps merge, green CI, all looked good.
Until users started complaining: “My orders aren’t going through.”
Turned out, the deployment file still pointed to an old URL for "Inventory Service"—a version we had already deprecated. The Git repo didn’t match reality.
env:
- name: INVENTORY_SERVICE_URL
value: "http://old-inventory-service.svc.cluster.local"
No crashes. No alarms. Just 200+ failed transactions we barely caught in time.
What went wrong?
Git had the wrong truth. Services don’t exist in a vacuum. They depend on each other—and those relationships change faster than code.
What we fixed:
- Started contract testing between services before deployment.
- Added staging checks to confirm live connectivity.
- Required interface definitions (OpenAPI + schema checks) in CI.
🧰 Tools: Where They Helped (and Hurt)
| Tool | Used For | Where It Broke Down |
|---|---|---|
| Git | Version control | No safeguards to block prod merges |
| Terraform | Infra as code | Too easy to delete things accidentally |
| Kubernetes | Orchestration | Amplified service failures across the board |
| FluxCD | GitOps operator | Applied bad configs with zero questions asked |
| Prometheus & Grafana | Monitoring | No alerts for silent data loss |
🧭 Takeaways
-
Declarative isn't always safe
Just because it's in code doesn't mean it's correct—or harmless. -
Git is not a gatekeeper
A merged PR doesn't mean things are good. It just means someone clicked "Approve." -
Automation needs adult supervision
Speed is great. But without checks? It's just a faster way to break stuff. -
Services need contracts
Without clear expectations between services, you’re setting yourself up for failure.
🚨 Final Thoughts
GitOps sounds beautiful on paper. Clean history. Reproducible infra. Automated everything.
But treat Git like the one source of truth? You’re one bad commit away from chaos.
We still use GitOps. But we’ve stopped treating it like magic. It’s more like power tools—great when used with care, dangerous when misused.
Be cautious. Add checks. Trust, but verify.