Finding and Stopping API Key Leaks Across Git Repositories

Finding and Stopping API Key Leaks Across Git Repositories

Reading time1 min
#devops#security#api#keys#leaks

Finding and Stopping API Key Leaks Across Git Repositories

A leaked key is rarely the result of one big mistake. It is a .env file that was not in .gitignore, a debug line in CI, or a token pasted into a test fixture. Once it is in a pushed commit, deleting the file does not help: it stays in history, in every clone and in every fork.

This post covers where secrets usually end up, how to scan for them, and the order of steps when one is found.

Where secrets end up

  • Committed config. .env, application.yml, terraform.tfvars, kubeconfigs, .npmrc with an auth token.
  • Git history. A file removed in a later commit is still readable in the old one.
  • CI logs. CI systems mask the values you register as secrets, but not values derived from them: base64 output, a URL with the token inside, set -x traces.
  • Container images. ARG and ENV values are stored in image metadata. A file copied in one layer and deleted in the next is still in the first layer.
  • Terraform state. Many resource attributes, including generated passwords, are stored in state in plain text.
  • Kubernetes manifests. A Secret in Git is base64, not encryption.
  • Frontend bundles and mobile apps. Anything shipped to a client is public.
  • Chat, tickets and wikis. Not scanned by repo tools at all.

For public repositories, assume a pushed key is found quickly. Automated scanners watch public commits all the time. GitHub also scans public repositories and public npm packages and sends matches to the token issuer through its secret scanning partner program, and push protection for users is on by default for pushes to public repositories.

Scan the history you already have

gitleaks scans Git history with git log -p. Since v8.19 the commands are git, dir and stdin; the older detect and protect still work but are hidden. The project is now in security-fix-only mode, so check its README for the current status before you standardise on it.

# full history of the current repo, secrets redacted in output
gitleaks git -v --redact --report-format json --report-path gitleaks.json .

# a directory that is not a Git repo, e.g. an unpacked artifact
gitleaks dir ./build

On an old repository the first run will find things that are already rotated or are test data. Record them as a baseline, so CI fails only on new findings:

gitleaks git --report-path baseline.json .
gitleaks git --baseline-path baseline.json --report-path findings.json .

TruffleHog adds verification: for many detectors it calls the provider's API to check whether the credential is live. That makes triage much faster. It also means the scanner sends found credentials to those APIs, so decide whether that is acceptable for your environment.

trufflehog git file://. --results=verified,unknown
trufflehog github --org=my-org --results=verified
trufflehog docker --image registry.example.com/my-app:1.4.2 --results=verified

Scanning container images matters because many leaks never touch Git.

Block new leaks before they are pushed

A pre-commit hook catches most mistakes on the developer's machine:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.30.1
    hooks:
      - id: gitleaks

The hook runs gitleaks git --pre-commit --redact --staged --verbose. Hooks are local and anyone can skip them, so run a secret scan in CI on every pull request too:

trufflehog git file://. --since-commit main --branch HEAD \
  --results=verified,unknown --fail

With --fail, TruffleHog exits with code 183 when it finds results. For repository-level enforcement on GitHub, enable push protection for the repository, which requires GitHub Secret Protection on private repositories.

Stop creating long-lived secrets

Scanning finds leaks. Fewer static secrets means fewer things to leak.

  • CI to cloud with OIDC. Instead of storing cloud keys in CI, let the pipeline exchange its OIDC token for short-lived credentials:
permissions:
  id-token: write
  contents: read
steps:
  - uses: aws-actions/configure-aws-credentials@v6
    with:
      role-to-assume: arn:aws:iam::123456789012:role/my-app-deploy
      aws-region: eu-central-1
  • Build secrets that do not end up in layers. Use BuildKit secret mounts:
# syntax=docker/dockerfile:1
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci
docker build --secret id=npmrc,src=$HOME/.npmrc -t my-app .
  • Runtime secrets from a secret manager. Vault, AWS Secrets Manager, Google Secret Manager or Azure Key Vault, read at startup or through the External Secrets Operator in Kubernetes. Do not write the secret value into Terraform code. If Terraform must handle it, use ephemeral resources (Terraform 1.10+) or write-only arguments (1.11+) where the provider supports them, and treat state as sensitive either way.
  • Scoped and short-lived tokens. A read-only token for one repository does far less damage than an organisation admin token.

When a secret is found

  1. Revoke or rotate first. Removing it from Git does not un-leak it. If rotation breaks something, that is still better than a live leaked key.
  2. Check whether it was used. Look at the provider's audit log for the key: CloudTrail for AWS access keys, the audit log for GitHub tokens, and so on. Look from the time of the first commit, not from when you noticed.
  3. Fix the source. Move the value to a secret manager and change the code to read it from there.
  4. Clean history only if it is worth it. git filter-repo can rewrite history, but forks, clones and caches keep the old commits. It is cleanup, not remediation.
  5. Add a rule. If the scanner missed the format, add a custom rule to .gitleaks.toml so it is caught next time.

Checklist

  • .env, key files and local config in .gitignore, with a committed .env.example instead.
  • gitleaks or TruffleHog as a pre-commit hook and as a required CI check.
  • Full-history scan of every repository once, with a baseline for old findings.
  • Image scans for secrets, not only for CVEs.
  • OIDC federation instead of static cloud keys in CI.
  • BuildKit secret mounts instead of ARG for build-time credentials.
  • A written rotation runbook for every key type you issue.