Securing Kubernetes Pods with OPA Gatekeeper

A pod spec can ask for a lot: privileged mode, host namespaces, host paths, root user, any image from anywhere. Kubernetes accepts all of it unless something in admission says no. This post shows how to use OPA Gatekeeper for that, and where it fits next to the built-in Pod Security Admission.

Start with Pod Security Admission

PodSecurityPolicy was removed in Kubernetes 1.25. Its built-in replacement is Pod Security Admission, which applies the Pod Security Standards (privileged, baseline, restricted) per namespace through labels:

kubectl label namespace my-app \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted

That covers privileged containers, host namespaces, host paths, capabilities, running as root and seccomp. Use it first. It is built in, fast and needs nothing to maintain.

Gatekeeper is for what PSA cannot express:

• Allowed image registries.
• Required CPU and memory limits.
• Required labels or annotations.
• Exceptions narrower than a whole namespace.
• Rules on objects other than pods (Ingress hosts, Service types and so on).

How Gatekeeper works

Gatekeeper runs as a validating admission webhook plus an audit controller. Policies come in two parts:

• A ConstraintTemplate holds the Rego logic and defines a new CRD kind.
• A Constraint is an instance of that kind: which objects it applies to, its parameters and its enforcement action.

The audit controller periodically checks existing objects against all constraints and writes violations into each constraint's status. That catches resources created before the policy existed.

Install

Pick a release from the Gatekeeper releases page and install it with Helm:

helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm install gatekeeper gatekeeper/gatekeeper \
  --namespace gatekeeper-system --create-namespace
kubectl -n gatekeeper-system get pods

You should see the controller manager pods and the audit pod running.

A template: allowed image registries

The rego field in a ConstraintTemplate uses Rego v0 syntax. Gatekeeper 3.19 and later can also accept Rego v1 through the code field with source.version: "v1". The example below sticks to the classic form, which matches the Gatekeeper policy library.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedrepos
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRepos
      validation:
        openAPIV3Schema:
          type: object
          properties:
            repos:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedrepos

        violation[{"msg": msg}] {
          container := input_containers[_]
          not image_allowed(container.image)
          msg := sprintf("container <%v> uses image <%v>, which is not from an allowed registry", [container.name, container.image])
        }

        image_allowed(image) {
          startswith(image, input.parameters.repos[_])
        }

        input_containers[c] {
          c := input.review.object.spec.containers[_]
        }
        input_containers[c] {
          c := input.review.object.spec.initContainers[_]
        }
        input_containers[c] {
          c := input.review.object.spec.ephemeralContainers[_]
        }

Two details matter here. The rule checks init and ephemeral containers, not only containers, because those are the obvious way around a check that looks at containers only. And the "is it allowed" test is a separate rule: the image is allowed if it matches any prefix. Writing not startswith(image, repos[_]) inline means "not matching some prefix", which flags every image as soon as there is more than one allowed registry.

A constraint in dry run

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: allowed-registries
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]
  parameters:
    repos:
      - "registry.example.com/"

enforcementAction accepts deny (the default), dryrun and warn. With dryrun, nothing is blocked and the audit controller records what would be. End each registry prefix with /, otherwise registry.example.com also matches registry.example.com.attacker.net/image.

Read audit results

kubectl get k8sallowedrepos allowed-registries -o yaml

Look at status.totalViolations and status.violations. The list is capped (20 entries per constraint by default, configurable with --constraint-violations-limit), so use totalViolations for the real count and the Gatekeeper metrics for trends. Fix the violations or agree on exceptions, then switch to warn and finally deny.

Test policies before they reach a cluster

The gator CLI evaluates templates and constraints offline.

# check manifests against templates and constraints
gator test -f manifests/ -f policies/

# run a suite of expected results
gator verify policies/...

A suite file lists cases with expected outcomes:

kind: Suite
apiVersion: test.gatekeeper.sh/v1alpha1
tests:
  - name: allowed-repos
    template: k8sallowedrepos-template.yaml
    constraint: allowed-registries.yaml
    cases:
      - name: internal-image
        object: samples/pod-internal.yaml
        assertions:
          - violations: no
      - name: docker-hub-image
        object: samples/pod-dockerhub.yaml
        assertions:
          - violations: yes

Run gator verify in the pipeline of the policy repository, so a broken template never reaches a cluster.

Use the library

Before writing Rego, check the Gatekeeper policy library. It has tested templates for allowed repositories, required labels, container limits, and a set that maps the old PodSecurityPolicy controls one by one. Those are useful if you need finer exceptions than PSA namespace labels allow.

Rules that match only pods

A constraint on Pod rejects pods created by a Deployment controller, not the Deployment. kubectl apply succeeds and the error appears in the ReplicaSet's events. Gatekeeper's expansion feature (ExpansionTemplate) can validate the pod template inside workloads at admission time, so the error reaches the person who applied the manifest.

Operational notes

• Gatekeeper is in the admission path. Run several replicas, watch webhook latency and decide on its failure policy deliberately.
• Keep constraints in Git and deploy them like any other code.
• Every constraint needs an owner and a reason; delete the ones nobody can explain.

Checklist

• Pod Security Admission on every namespace first.
• Gatekeeper only for rules PSA cannot express.
• Templates check containers, init containers and ephemeral containers.
• New constraints start with dryrun, then warn, then deny.
• Audit results reviewed before enforcement.
• gator verify in CI for every policy change.