When GitOps Deletes Things: Pruning, Cascades and Guardrails

A GitOps controller makes the cluster match Git. That includes deleting whatever is no longer in Git. This is the feature that keeps clusters clean, and it is also how a refactoring pull request removes a production namespace.

This post covers the common ways a declarative change turns into a delete, and the settings in Argo CD, Flux and Kubernetes that limit the damage.

How a change becomes a delete

The rendered output shrinks. A folder gets moved, an Application points at a new path, an overlay stops including a base, a Helm value disables a subchart. The YAML is valid and CI is green. With pruning enabled, everything that is no longer rendered gets deleted.

The parent object is deleted. Deleting an Argo CD Application that carries the resources-finalizer.argocd.argoproj.io finalizer deletes everything it manages. With ApplicationSets it is one step further away: remove an entry from a generator (a cluster from a list, a directory from a Git generator) and the generated Application goes away, along with its workloads. In Flux, deleting a Kustomization with prune: true garbage-collects every object it applied.

Kubernetes cascades. Deleting a Namespace deletes everything in it. Deleting a CRD deletes every custom resource of that kind. Deleting a PVC whose PersistentVolume has reclaim policy Delete (the default for dynamically provisioned volumes) deletes the disk.

Infrastructure controllers. With Crossplane or a Terraform controller, a removed manifest can mean a deleted cloud database, not just a deleted pod.

Self-heal during an incident. Someone scales a Deployment up by hand to absorb load, and the controller scales it back down a few minutes later. Nothing is deleted, but the effect is the same.

Argo CD guardrails

Keep allowEmpty at its default (false). Automated sync with prune then refuses to prune when the application renders zero resources. It does not help when the output only shrinks.

Protect individual resources with sync options:

metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false,Delete=false

Prune=false keeps the resource when it disappears from Git. Delete=false keeps it when the Application is deleted. Prune=confirm and Delete=confirm are a middle ground: the operation waits until someone confirms it in the UI or CLI.

For ApplicationSets, stop the controller from deleting Applications, and keep resources if an Application is deleted anyway:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-app
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - list:
        elements:
          - cluster: staging
            url: https://staging.example.com
          - cluster: production
            url: https://production.example.com
  syncPolicy:
    applicationsSync: create-update
    preserveResourcesOnDeletion: true
  template:
    metadata:
      name: 'my-app-{{.cluster}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/example/deploy.git
        targetRevision: main
        path: 'apps/my-app/{{.cluster}}'
      destination:
        server: '{{.url}}'
        namespace: my-app

With create-update, removing an element leaves the Application in place for a human to delete.

Flux guardrails

Mark objects that must survive with kustomize.toolkit.fluxcd.io/prune: disabled. Flux will apply them but never garbage-collect them.

Set a deletion policy on Kustomizations that manage stateful things:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: databases
  namespace: flux-system
spec:
  interval: 10m
  path: ./infrastructure/databases
  prune: true
  deletionPolicy: Orphan
  sourceRef:
    kind: GitRepository
    name: flux-system

The default, MirrorPrune, deletes managed objects with the Kustomization when prune is true. Orphan leaves them in the cluster.

During an incident, flux suspend kustomization my-app stops both new revisions and drift correction until flux resume.

Kubernetes and cloud guardrails

• Use reclaimPolicy: Retain on StorageClasses for data volumes, or patch existing PVs to persistentVolumeReclaimPolicy: Retain.
• Keep CRDs and namespaces in their own Application or Kustomization, separate from the workloads that use them.
• Enable deletion protection on cloud databases and keep backups that do not live in the same cluster or account.
• Block deletes of critical objects at admission time. A ValidatingAdmissionPolicy stops humans and controllers alike:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: protect-namespaces
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["DELETE"]
        resources: ["namespaces"]
  validations:
    - expression: >-
        !(has(oldObject.metadata.labels) &&
          'protected' in oldObject.metadata.labels &&
          oldObject.metadata.labels['protected'] == 'true')
      message: "namespace is labeled protected=true"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: protect-namespaces
spec:
  policyName: protect-namespaces
  validationActions: [Deny]

To delete a protected namespace on purpose, remove the label first. That is one extra step, and it is visible.

Show deletions in review

Reviewers see the diff of the YAML source, not the diff of what will be rendered. Render both sides in CI and list objects that disappear:

#!/usr/bin/env bash
set -euo pipefail
path=apps/my-app/production
git worktree add -q /tmp/base origin/main
list() {
  kustomize build "$1" \
    | yq -N '[.kind, .metadata.namespace // "", .metadata.name] | join("/")' | sort
}
comm -23 <(list "/tmp/base/$path") <(list "$path") > removed.txt
git worktree remove --force /tmp/base
if [[ -s removed.txt ]]; then
  echo "This change removes objects:"; cat removed.txt
  exit 1
fi

Make the job required, and let a maintainer override it when the removal is intended. For Helm, replace kustomize build with helm template. argocd app diff and flux diff kustomization show the same thing against the live cluster.

Checklist

• Pruning is on only where you understand what it will delete.
• Stateful objects carry Prune=false,Delete=false (Argo CD) or prune: disabled (Flux).
• ApplicationSets use create-update and preserveResourcesOnDeletion.
• Flux Kustomizations with data use deletionPolicy: Orphan.
• PVs for data use Retain, cloud databases have deletion protection and outside backups.
• An admission policy blocks deletion of protected namespaces.
• CI lists removed objects on every pull request.
• Everyone on call knows how to pause sync.