Hardcoded Secrets Sprawl: How to Find, Rotate, and Prevent Credential Leaks at Scale

In 2025, researchers discovered over 29 million hardcoded secrets in public and private repositories — a 34% year-over-year increase. API keys, database passwords, OAuth tokens, and cloud credentials left directly in source code are not a new problem. They are, however, an accelerating one. The rise of AI-assisted coding tools has added a new wrinkle: developers accepting generated code that embeds insecure credential patterns absorbed from training data. This playbook covers how to find every secret in your environment, triage what poses active risk, respond to live exposures, and build the tooling and culture to stop the pattern at source.

The Challenge

Hardcoded secrets are not simply a developer error — they are a systemic failure that compounds across every dimension of a modern software organisation. Understanding the full scope is prerequisite to fixing it.

• Scale that defeats manual review. At 29 million secrets discovered in 2025 alone, the volume far exceeds any team's ability to find and remediate manually. Automated scanning is non-negotiable.
• Git history is permanent. A secret deleted from the current branch still lives in every prior commit, every fork, and every clone of that repository. A standard code review catches nothing once a commit lands.
• AI code generation amplifies the problem. Code assistants trained on public repositories have ingested millions of files containing real credentials. They surface patterns that look correct and compile cleanly — but may reflect insecure credential handling that developers accept without scrutiny.
• Supply chain propagation. A single leaked credential in a dependency's CI/CD pipeline or build artifact can compromise every downstream consumer. Attackers specifically target the build layer because blast radius is maximised.
• Legacy codebases resist change. Applications with ten-year histories routinely contain credentials no one currently knows the purpose of — owned by services that may no longer exist, or may still be fully active.

Why It Remains Unsolved

The technical solution is well-understood. The operational gap is why this problem persists at scale across organisations of every size.

• Detection outpaces remediation. Scanning tools surface thousands of findings faster than security or development teams can process them. Without triage and prioritisation, teams lose confidence in the tool output and stop acting on alerts.
• Developers embed secrets for convenience. Local development, quick prototypes, and deadline pressure create persistent incentives to skip the extra step of vault lookup or environment variable configuration.
• Rotation is operationally expensive. Rotating a single high-value credential requires identifying every service and pipeline that depends on it, coordinating restarts, updating configurations in multiple environments, and verifying nothing broke. Multiplied by hundreds of findings, most teams defer indefinitely.
• No universal standard for secret storage. Even within a single organisation, teams may use five different approaches: environment variables, AWS Secrets Manager, HashiCorp Vault, Kubernetes secrets, and config files. Inconsistency slows automation.
• Blame culture suppresses disclosure. When developers fear consequences for having committed a secret, they silently delete the line and hope no one notices — instead of triggering the rotation workflow. The credential remains live and unrotated.

Step-by-Step Resolution Framework

Step 1: Secret Discovery Audit

You cannot remediate what you have not found. A comprehensive discovery audit covers five surfaces: source code repositories, CI/CD pipeline configurations and logs, container images, infrastructure-as-code, and application logs.

Repository scanning with Gitleaks:

# Scan full git history of current repository
gitleaks detect --source . \
  --log-opts="--all" \
  --report-format json \
  --report-path gitleaks-report.json

# Scan a specific path including all branches
gitleaks detect --source /path/to/repo \
  --log-opts="--all --branches" \
  --report-format sarif \
  --report-path results.sarif

# Use a custom config to tune rules and reduce false positives
gitleaks detect --source . \
  --config .gitleaks.toml \
  --log-opts="--all"

Deep history scanning with TruffleHog:

# Scan a GitHub organisation for secrets in all repos
trufflehog github \
  --org=your-org-name \
  --token=$GITHUB_TOKEN \
  --only-verified \
  --json 2>&1 | tee trufflehog-org-scan.json

# Scan a local git repo with full history
trufflehog git file://. \
  --only-verified \
  --json

# Scan a Docker image layer by layer
trufflehog docker \
  --image=your-registry/your-image:tag \
  --json

Scanning CI/CD artifacts and container images:

# Export and inspect a container filesystem
docker save your-image:tag | \
  tar xO | \
  trufflehog filesystem /dev/stdin

# Scan Kubernetes secrets for plaintext values
kubectl get secrets --all-namespaces -o json | \
  jq '.items[] | {name:.metadata.name, ns:.metadata.namespace, data:.data}' | \
  base64 -d 2>/dev/null

GitGuardian is the recommended SaaS layer for continuous monitoring across an entire organisation's GitHub or GitLab estate. It provides real-time alerts on pushes containing secrets, validity checking against provider APIs, and a remediation dashboard that tracks resolution status. Run it alongside local scanning rather than instead of it.

Step 2: Triage and Risk Assessment

Raw scanner output is noise until it is prioritised. The critical dimension is not how many secrets you found — it is which secrets are currently active and what damage their misuse would cause.

Triage decision tree:

• Is the credential still valid? Test programmatically where possible (AWS STS GetCallerIdentity, GitHub token introspection, Stripe key validation). Invalid credentials are low priority — document and close.
• What is the blast radius? A database root password with no network egress is different from an AWS access key with AdministratorAccess. Score each finding by: access scope, data sensitivity of systems reachable, whether the credential is rotatable without downtime.
• Is it in a public repository? Any secret ever pushed to a public repo must be treated as fully compromised regardless of deletion — it was indexed by automated harvesters within seconds of the push.
• How old is the exposure? A credential committed three years ago in a public repo has a very high probability of already being abused. Pivot immediately to access log review rather than extended analysis.
• Is it in git history vs. current HEAD? Both require rotation, but history-only findings allow more coordination time than live credentials in the active codebase.

Assign each finding to one of three tracks: P1 — Rotate Immediately (live, valid, high blast radius); P2 — Rotate This Sprint (valid but limited scope, or historical in private repos); P3 — Track and Suppress (invalid, test credentials, or intentional fixtures with documented rationale).

Step 3: Incident Response for Live Secrets

When a valid high-impact credential is confirmed exposed, treat it as an active security incident. Time from detection to revocation is the only metric that matters in the first hour.

Leaked AWS access key — step-by-step response:

# 1. IMMEDIATELY deactivate the key (do not delete yet - preserve for forensics)
aws iam update-access-key \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive \
  --user-name affected-service-account

# 2. Review CloudTrail for all activity on this key
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time 2026-01-01T00:00:00Z \
  --output json | jq '.Events[] | {time:.EventTime, event:.EventName, ip:.SourceIPAddress}'

# 3. Check for new IAM users, roles, or access keys created by this key
aws iam list-users --output json | jq '.Users[] | select(.CreateDate > "2026-01-01")'
aws iam list-roles --output json | jq '.Roles[] | select(.CreateDate > "2026-01-01")'

# 4. Look for unusual resource creation in all regions
for region in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  echo "=== $region ==="
  aws ec2 describe-instances --region $region \
    --query 'Reservations[].Instances[?LaunchTime>=`2026-01-01`]' --output text
done

# 5. After forensic review is complete, create a replacement key
aws iam create-access-key --user-name affected-service-account

# 6. Update all systems consuming the old key
# (use your secrets manager or configuration management tooling)

# 7. Delete the old key after confirming replacement is working
aws iam delete-access-key \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --user-name affected-service-account

Notification workflow: Alert the security team and service owner simultaneously — not sequentially. The service owner needs to prepare for the credential swap; the security team needs to begin the access log review. File an incident ticket immediately, even before analysis is complete. If personal data is accessible via the exposed credential and you operate under PIPEDA, GDPR, or equivalent regulation, begin the breach notification clock.

Step 4: Pre-Commit Prevention

Prevention at commit time stops the problem before it enters the repository. Pre-commit hooks provide the lowest-friction enforcement point because they run on the developer's machine before any code reaches a remote.

.pre-commit-config.yaml using Gitleaks and detect-secrets:

repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.21.2
    hooks:
      - id: gitleaks
        name: Detect hardcoded secrets
        description: Scan for secrets before commit
        entry: gitleaks protect --staged --redact --verbose
        language: golang
        pass_filenames: false

  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.5.0
    hooks:
      - id: detect-secrets
        name: detect-secrets
        args: ['--baseline', '.secrets.baseline']
        exclude: package-lock.json

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v5.0.0
    hooks:
      - id: check-added-large-files
        args: ['--maxkb=500']
      - id: check-merge-conflict
      - id: detect-private-key

Install across the team:

# Install pre-commit
pip install pre-commit

# Install hooks defined in .pre-commit-config.yaml
pre-commit install

# Run against all files once to establish baseline
pre-commit run --all-files

CI/CD pipeline scanning is the backstop for commits that bypass or predate pre-commit hooks. Add a Gitleaks scan step to every pull request pipeline:

# GitHub Actions example
- name: Run Gitleaks
  uses: gitleaks/gitleaks-action@v2
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}

IDE plugins: GitGuardian's VS Code extension and JetBrains plugin provide inline highlighting of detected secrets as developers type, before they even stage a file. These are worth mandating as part of your developer onboarding baseline.

Step 5: Vault Integration

Pre-commit prevention addresses new secrets. Vault integration addresses existing ones. The migration pattern is straightforward: replace every hardcoded credential reference with a runtime lookup, and store the actual secret in a managed secrets store.

Before (hardcoded — never do this):

import boto3

# INSECURE: hardcoded credentials in source code
s3 = boto3.client(
    's3',
    aws_access_key_id='AKIAIOSFODNN7EXAMPLE',
    aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    region_name='ca-central-1'
)

After — AWS Secrets Manager lookup:

import boto3
import json

def get_secret(secret_name: str, region: str = 'ca-central-1') -> dict:
    client = boto3.client('secretsmanager', region_name=region)
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response['SecretString'])

# Credentials retrieved at runtime, never stored in code
creds = get_secret('prod/myapp/s3-access-key')
s3 = boto3.client(
    's3',
    aws_access_key_id=creds['access_key_id'],
    aws_secret_access_key=creds['secret_access_key'],
    region_name='ca-central-1'
)

After — HashiCorp Vault KV lookup:

import hvac
import os

def get_vault_secret(path: str, key: str) -> str:
    client = hvac.Client(
        url=os.environ['VAULT_ADDR'],
        token=os.environ['VAULT_TOKEN']  # Token from env, not hardcoded
    )
    secret = client.secrets.kv.v2.read_secret_version(
        path=path,
        mount_point='secret'
    )
    return secret['data']['data'][key]

db_password = get_vault_secret('myapp/database', 'password')

After — Azure Key Vault with managed identity (no credentials at all):

from azure.identity import ManagedIdentityCredential
from azure.keyvault.secrets import SecretClient

# Managed identity: no credentials in code or environment at all
credential = ManagedIdentityCredential()
client = SecretClient(
    vault_url="https://your-vault.vault.azure.net/",
    credential=credential
)
db_conn_string = client.get_secret("prod-db-connection-string").value

The Azure Managed Identity pattern is the gold standard: the application has no credentials to leak because authentication is handled entirely by the platform identity layer.

Step 6: Automated Rotation

Manual rotation does not scale. Once credentials are in a vault, automated rotation on a schedule eliminates the operational burden and limits the window of exposure for any single credential.

HashiCorp Vault dynamic secrets for AWS:

# Configure the AWS secrets engine in Vault
vault secrets enable aws

vault write aws/config/root \
  access_key=$VAULT_AWS_ACCESS_KEY_ID \
  secret_key=$VAULT_AWS_SECRET_ACCESS_KEY \
  region=ca-central-1

# Define a role that generates scoped credentials on demand
vault write aws/roles/myapp-s3-role \
  credential_type=iam_user \
  policy_arns=arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
  default_ttl=1h \
  max_ttl=24h

# Application requests fresh credentials at startup or on TTL expiry
# These credentials never existed before the request and expire automatically
vault read aws/creds/myapp-s3-role

With dynamic secrets, there are no long-lived credentials to rotate — every set of credentials is ephemeral and scoped to the minimum required permission. Even if a credential is captured, it expires within the configured TTL.

AWS Secrets Manager automatic rotation with Lambda:

# Enable automatic rotation on an existing secret (90-day schedule)
aws secretsmanager rotate-secret \
  --secret-id prod/myapp/database \
  --rotation-lambda-arn arn:aws:lambda:ca-central-1:123456789012:function:RotateDbCredentials \
  --rotation-rules AutomaticallyAfterDays=90

# Verify rotation status
aws secretsmanager describe-secret \
  --secret-id prod/myapp/database \
  --query '{LastRotated:LastRotatedDate, NextRotation:NextRotationDate, RotationEnabled:RotationEnabled}'

Step 7: Developer Education and Culture

Tools prevent known-bad patterns. Culture prevents patterns the tools have not seen yet. The goal is a development team that instinctively reaches for a vault rather than a string literal.

Blameless disclosure: Publish a clear, written policy that any developer who self-reports a committed secret will face no punitive consequence, provided they report it promptly. This single policy change dramatically improves detection time. The alternative — blame culture — produces silent deletions that leave live credentials unrotated.

Quick-reference card (distribute to all developers):

• Never hardcode a secret, even temporarily. Use a placeholder env var from the start.
• Store local dev secrets in a .env file that is in .gitignore and never committed.
• Review all AI-generated code for credential patterns before accepting it.
• If you discover a secret in history: report to security immediately, do not just delete the line.
• The tools will catch most things. You are the last line of defence for the rest.

Secure-by-default project templates: Provide bootstrapped project templates for each language stack that include a pre-configured .pre-commit-config.yaml, a .gitignore excluding common secret files, and example vault integration code for your organisation's secrets manager. Developers who start from the template inherit the controls automatically.

Tips and Tricks

Quick Wins vs. Long-Term Fixes

Frequently Asked Questions

Do I need to rewrite git history to remove an exposed secret?

History rewriting via git filter-repo is time-consuming, disruptive, and incomplete — every fork and clone retains the old history. Rotate the credential immediately and treat the rewrite as a secondary cleanup step. On public repositories, assume the secret was indexed by automated harvesters within minutes of the original push. History rewriting is a hygiene measure, not a security control.

How do I handle secrets in container images?

Container image layers are permanent. A secret added in an early build layer and removed in a later layer still exists in the intermediate layer and can be extracted with docker history or direct layer inspection. The only fix is to rebuild the image without the secret ever having been present — and to rotate the exposed credential immediately. Use multi-stage builds and build-time secret injection via docker build --secret rather than ENV or ARG directives.

What about secrets in CI/CD environment variables?

CI/CD environment variables are significantly safer than source code but still require management. Secrets stored in CI/CD platform secret stores (GitHub Actions secrets, GitLab CI variables, Jenkins credentials) should still be rotated regularly, scoped to the minimum required permission, and audited for stale values. Log output should be scanned to confirm secrets are not being printed during builds — debug output and error messages frequently expose credential values.

How do we handle secrets in Terraform state files?

Terraform state files frequently contain plaintext values for any resource whose provider stores sensitive attributes — including database passwords, private keys, and API tokens. Store state remotely in an encrypted backend (S3 with encryption at rest and strict IAM, Terraform Cloud with sensitive value masking), restrict access to state files to the CI/CD pipeline service account only, and use sensitive = true on all sensitive output values to prevent them appearing in plan output.