AWS access key committed to Git history: rotate first, then clean the repo

On a Thursday morning a developer pushed a branch to our private GitHub repository for the reporting service. Eleven minutes later our security channel received a secret scanning alert, and almost at the same time AWS emailed the account's root contact saying an access key had been found in a public location. The repository was private, so the second email confused us for a minute: the same branch had also been pushed to a personal fork the developer used for a side experiment, and that fork was public.

The commit had added .env.local, which the developer had copied from a teammate. It contained:

AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
REPORTS_BUCKET=reports-prod-example

(Values above are AWS's documented example credentials, not the real ones.) The next commit deleted the file, and the developer assumed that was the end of it. It was not: any clone, fork or cached view of the earlier commit still had the key.

Reproducing it

The mistake is easy to repeat, which is part of why we wrote this up:

1. .gitignore listed .env but not .env.local or .env.*.
2. git add . picked up the new file.
3. A follow-up git rm .env.local removed it from the tip, but git log -p --all -- .env.local still printed the full contents.

git log --all --oneline -- .env.local
git show <first-commit>:.env.local

What we ruled out

• "The repo is private, so it is fine." Private repositories get cloned onto laptops and CI runners, and forks can be public. Once a secret leaves the secrets manager, we assume it is known.
• "We deleted it in the next commit." Git stores snapshots; a later commit does not change the earlier one. Only a history rewrite removes it, and even that cannot reach existing clones.
• "Rewrite history first, rotate later." Rewriting takes coordination with everyone who has a clone. Rotating takes two minutes and makes every copy of the key worthless. Order matters.

Root cause

The key belonged to a long-lived IAM user created two years earlier for a nightly export job. It had s3:* on the reports bucket and had been shared through chat and copied into local env files for debugging. Three things lined up: a static credential that should not have existed, a .gitignore pattern that was too narrow, and no secret scanning before the push left the laptop.

What is happening under the hood

A Git commit points at a tree, which points at blobs. Deleting a file creates a new tree without that blob, but the old commit and blob remain reachable from the branch history. Pushing sends every object the remote does not yet have, so the blob reaches the server and every future clone. Hosting platforms also keep pull request refs and may cache diffs, which is why GitHub's guidance includes contacting support after a rewrite.

Public repositories are scanned continuously by attackers, not just by vendors. AWS and GitHub run partner programs that detect AWS key formats and notify the owner, but those notifications are a race against automated scrapers, not a safety net.

Environment

Minimal reproduction

git init leak-demo && cd leak-demo
printf 'API_KEY=sk_test_xxx\n' > .env.local
git add . && git commit -m "add config"
git rm .env.local && git commit -m "remove config"
git log -p --all -S 'sk_test_xxx'   # the secret is still in history

Who is affected

Any repository where a credential was ever committed, on any branch or tag that was pushed, including merged and deleted branches whose commits were merged. Squash merges hide it from the main branch, but the original branch and its pull request ref may still hold it.

The fix

We worked in this order and timed each step in the incident doc.

1. Revoke and replace the credential (minute 0 to 9). We created a replacement key for the export job, stored it in AWS Secrets Manager, deployed the job, then deactivated and deleted the leaked key:

aws iam update-access-key --user-name reports-export \
  --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive
aws iam delete-access-key --user-name reports-export \
  --access-key-id AKIAIOSFODNN7EXAMPLE

2. Audit what the key did (minute 9 to 40). CloudTrail lookups by access key covered management events; S3 data events were enabled on the bucket, so object reads were there too:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time 2025-01-02T00:00:00Z --query 'Events[].[EventTime,EventName,Username]'

We found two ListBuckets calls from an unfamiliar IP range about six minutes after the fork push, both denied, because the policy did not allow that action. No object reads came from anywhere but the export job's known address.

3. Clean the history. We deleted the public fork, then rewrote the private repository with git-filter-repo on a fresh mirror clone. Version 2.47 added --sensitive-data-removal, which fetches all refs (not only branches and tags) so they get cleaned too, and reports the first changed commits and any orphaned LFS objects:

git clone --mirror git@github.com:example-org/reporting.git
cd reporting.git
git filter-repo --sensitive-data-removal --invert-paths --path .env.local
git push --force --mirror origin

For secrets that were pasted into tracked source files rather than a separate file, we use --replace-text with an expressions file instead, which rewrites the matching string in every blob:

regex:AKIA[0-9A-Z]{16}==>REMOVED_AWS_KEY_ID
literal:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY==>REMOVED_SECRET

git filter-repo --sensitive-data-removal --replace-text ../expressions.txt

We then opened a GitHub Support request with the first changed commit hashes so cached views and pull request refs could be removed, and asked the six people with clones to re-clone rather than pull, because a pull and push from an old clone reintroduces the objects.

4. Remove the static credential altogether. The export job moved to an IAM role assumed through its ECS task role, so no long-lived key exists to leak next time.

Alternatives and trade-offs

• Skip the rewrite. If the secret is revoked and the repo is private, rotating alone may be acceptable; GitHub's own documentation says as much. We rewrote because a public fork had existed and we wanted the alerts closed cleanly.
• BFG Repo-Cleaner. Still works, but it is no longer the tool GitHub documents, and filter-repo handles refs and reporting better.
• Make the repo history "append-only" and never rewrite. Some teams forbid force pushes on principle; then rotation and monitoring are the whole response.

How to test the fix

# On a fresh clone: no blob in any ref should contain the key pattern.
git clone git@github.com:example-org/reporting.git verify && cd verify
git log --all -p | grep -E 'AKIA[0-9A-Z]{16}' && echo "STILL PRESENT" || echo "clean"
gitleaks git -v .

gitleaks git scans every commit reachable in the repository and exits non-zero on a finding, so it works as a CI check too.

Confirming in production

• The export job ran that night with the role-based credentials and produced the usual 212 files.
• IAM credential report showed zero active access keys for reports-export.
• GitHub's secret scanning alert was closed as "revoked", and the AWS notification case was answered with the rotation timestamp.
• Two weeks of CloudTrail showed no further calls with the old key ID (all would fail anyway).

Regression guard

• .gitignore now uses .env* with !.env.example.
• A pre-commit hook runs gitleaks on staged changes for every developer:

repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.22.1
    hooks:
      - id: gitleaks

• Secret scanning push protection is enabled for the organization's repositories, so known credential formats are blocked on push, not just alerted on afterwards.
• A CI job runs gitleaks git over the full history on the default branch weekly.
• An AWS Config rule flags IAM access keys older than 90 days.

Security implications

The leak exposed a credential with write access to report data. The response limited real exposure to two denied API calls. The history rewrite reduced future discovery but cannot guarantee removal from clones made before it, which is why rotation came first. Replacing the static key with a role removed the whole class of risk for that job.

Prevention checklist

• Rotate first, audit second, clean history third.
• Prefer short-lived role credentials over static keys; inventory and expire the rest.
• Ignore .env* by pattern, and keep a committed .env.example with placeholders.
• Run secret scanning both before commit and on push.
• Do not share secrets through chat; share a path in the secrets manager.

References

• GitHub Docs: Removing sensitive data from a repository — https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository
• GitHub Docs: About push protection — https://docs.github.com/en/code-security/secret-scanning/introduction/about-push-protection
• git-filter-repo project and manual — https://github.com/newren/git-filter-repo
• AWS IAM User Guide: Manage access keys for IAM users — https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
• Gitleaks — https://github.com/gitleaks/gitleaks