Secret scanning for AI-generated code: why your diff might be leaking API keys

AI coding agents are fast, creative, and sometimes dangerously helpful. One of the most overlooked risks when reviewing AI-generated code is secret leakage — hardcoded API keys, tokens, passwords, and credentials that the agent embeds directly into your source files, config files, or environment variables.

This is not a theoretical concern. Every week, developers push AI-generated diffs that contain real credentials to public repositories. Automated secret scanners catch some of these, but many slip through — especially in private repositories, internal tools, and configuration files that never reach public CI.

This article explains why AI coding agents leak secrets, what kinds of secrets appear most often, how to scan for them before merging, and how to build a lightweight secret-scanning gate that works for solo developers and small teams.

Why AI coding agents leak secrets

AI coding agents do not „know” that a string is sensitive. They pattern-match from training data, which includes millions of real configuration files, environment variable assignments, and connection strings. When you ask an agent to „set up a database connection” or „add authentication,” it frequently produces code like this:

# What the agent generates
DATABASE_URL = "postgres://admin:MyS3cretP@ss@db.example.com:5432/prod"
API_KEY = "sk-proj-abc123..."
STRIPE_SECRET = "sk_live_51N..."

These are not fake placeholders. They are real patterns that agents reproduce from training data, sometimes with actual keys the agent picked up from your project context, environment files, or previous conversations.

Here are the most common scenarios:

• Copying from .env files — agents read your local environment and reproduce values into source files
• Generating realistic-looking credentials — the agent creates plausible API keys, tokens, and passwords that follow real format patterns
• Hardcoding connection strings — database URLs, Redis endpoints, and S3 paths with embedded credentials
• Creating config files with real secrets — YAML, JSON, TOML configs that contain tokens, private keys, and passwords
• Mirroring test fixtures — test data that contains „example” keys that happen to follow valid formats

The six types of secrets AI agents leak

Based on analysis of thousands of AI-generated diffs, here are the categories you should scan for:

1. API keys and tokens

The most common leak. AI agents frequently generate code that includes:

• OpenAI, Anthropic, and other AI provider API keys
• Cloud provider access keys (AWS, GCP, Azure)
• Service-to-service tokens (Stripe, SendGrid, Twilio)
• OAuth client secrets

Pattern example: sk-proj-..., sk_live_..., AKIA...

2. Database connection strings

Agents love to include complete connection URLs with embedded credentials:

postgres://user:password@host:5432/db
mysql://root:password123@localhost/mydb
mongodb+srv://admin:pass@cluster.mongodb.net/db

3. Private keys and certificates

RSA, SSH, and PEM keys generated or referenced by agents in config files, deployment scripts, or test fixtures.

4. OAuth and JWT secrets

Client secrets, signing keys, and JWT secrets that agents embed in authentication configuration blocks.

5. Internal URLs and endpoints

Not secrets per se, but internal API endpoints, admin panels, and staging URLs that should not be in source control.

6. Credential patterns in configuration

YAML, JSON, and TOML files with password fields, token fields, and secret fields that the agent fills with realistic-looking values.

How to scan AI diffs for secrets before merge

You need a systematic approach — not just „I’ll be careful.” Here is a practical five-step secret-scanning workflow:

Step 1: Always review the diff, not just the final file

Secrets often appear in the diff even when the final file looks clean — because the agent adds a credential in one iteration and removes it in the next, or because the diff shows environment variable assignments that the agent later refactors. Always scan the unified diff.

Step 2: Scan for the six secret categories

Run a pattern-based scan on every AI-generated diff before merging. At minimum, check for:

• API key patterns (OpenAI, AWS, Stripe, generic key-value pairs)
• Database connection strings with embedded credentials
• Private key blocks (BEGIN RSA, BEGIN PRIVATE KEY)
• OAuth/JWT secret patterns
• Password fields in config files (YAML, JSON, TOML, .env)
• Internal URL patterns

Tools like gitleaks, truffleHog, and Snyk can scan repositories, but they typically run on the full repo — not on the specific AI-generated diff. For AI code review, you want a diff-focused scanner that checks only what changed.

Step 3: Flag config file changes separately

AI agents modify configuration files more often than business logic files. Any diff that touches .env, config.yml, docker-compose.yml, settings.json, or similar files deserves extra scrutiny — even if no secret patterns are detected, because agents sometimes encode credentials in non-obvious ways.

Step 4: Check for „example” secrets that follow real patterns

AI agents often generate „placeholder” secrets that follow real format patterns:

# "Fake" but realistic-looking
API_KEY = "sk-proj-1234567890abcdef1234567890abcdef"
DB_PASSWORD = "MyPassword123!"

These are risky because:

• They follow valid formats and can be confused with real keys
• Secret scanners will flag them, creating noise in your security pipeline
• Other developers may assume they are real and use them
• Some „example” keys are actually valid keys from training data

Best practice: replace all agent-generated credentials with proper secret references (os.environ["API_KEY"], ${DB_PASSWORD}, vault references) before committing.

Step 5: Integrate secret scanning into your review workflow

Make secret scanning a required step in your AI code review process:

1. Before merge: Run a diff-focused secret scanner on every AI-generated pull request
2. In CI: Add a secret-scanning gate that fails the build if secrets are detected
3. Pre-commit: Use a pre-commit hook that scans staged changes for secret patterns
4. Periodic audit: Run a full-repository scan weekly to catch secrets that slipped through

Building a CI secret-scanning gate

Here is a minimal GitHub Actions workflow that scans AI-generated diffs for secrets:

name: Secret Scan on AI-Generated Changes
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Get diff
        run: |
          git diff origin/main...HEAD > /tmp/ai-diff.txt
      - name: Scan diff for secrets
        run: |
          python3 scripts/scan_diff_secrets.py /tmp/ai-diff.txt
          # Exit code 1 if secrets found
      - name: Upload scan report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: secret-scan-report
          path: secret-scan-report.json

This pattern works with any diff-focused secret scanner. The key is scanning the diff (what changed) rather than the entire repository.

What to do when a secret is found in an AI diff

If your scan finds a secret in an AI-generated change:

1. Do not merge the PR — even if the secret looks like a placeholder
2. If the secret is real: rotate it immediately, scan the agent’s context for other leaks, and check git history for accidental commits
3. If the secret is a realistic placeholder: replace it with a proper secret reference before merging
4. Add an allowlist entry only if you are certain the pattern is a test fixture that will never be real
5. Document the finding in your review notes so the team learns what the agent tends to leak

Common anti-patterns in AI secret handling

Anti-pattern 1: „It is just a test fixture”

AI-generated test fixtures frequently contain realistic-looking secrets. The problem: these fixtures end up in public repos, and automated scanners flag them. Worse, some „test” keys are actually valid keys from training data.

Fix: Use clearly fake patterns (test-key-not-real) or environment variable references in test fixtures.

Anti-pattern 2: „I will clean it up later”

The agent adds a real API key „temporarily” and you plan to replace it. Later never comes. The key stays in git history forever.

Fix: Never commit secrets, even temporarily. Use os.environ references from the start.

Anti-pattern 3: „The scanner found nothing”

Secret scanners have false negatives. They miss encoded secrets, custom formats, and multi-line keys. A clean scan does not mean the diff is safe.

Fix: Combine automated scanning with manual review of config file changes and credential-related code.

Anti-pattern 4: „It is in .env.example, not .env”

Agents sometimes put real values in .env.example or .env.template files, which are typically not gitignored. These files end up in your repository with real credentials.

Fix: Only put placeholder references (DATABASE_URL=your_db_url_here) in example files. Never include real values.

Anti-pattern 5: „The agent refactored it out”

An agent may add a secret in one change and refactor it to an environment variable in the next. The secret still exists in the diff and git history.

Fix: Always scan the full diff history of an AI-assisted branch, not just the final state. If a secret was ever committed, rotate it.

How CodeRiskTools helps with secret scanning

If you want a structured, diff-focused approach to catching secrets in AI-generated code:

• Developer Safety Kit withdrawn from public sale; existing buyer access retained.
• Safety Kit retired — current catalog adds expanded security prompts, risk scoring, and client-ready documentation for security findings
• CodeRiskTools Scanner 3.0.0 is the current public scanner: MIT-licensed and free on GitHub for local secret and configuration-change review.

Each tool focuses on a specific part of the secret-scanning problem — from review checklists to automated diff scanning to CI integration.

Secret scanning checklist for AI-generated code

Use this checklist before merging any AI-generated pull request:

• ☐ Have I reviewed the diff (not just the final files)?
• ☐ Have I scanned for API key patterns (OpenAI, AWS, Stripe, generic)?
• ☐ Have I checked for database connection strings with embedded credentials?
• ☐ Have I searched for private key blocks (RSA, SSH, PEM)?
• ☐ Have I reviewed all config file changes (.env, YAML, JSON, TOML)?
• ☐ Have I replaced all agent-generated credentials with environment variable references?
• ☐ Have I flagged any „example” secrets that follow real patterns?
• ☐ If a real secret was found: have I rotated it and checked git history?
• ☐ Is my CI pipeline configured to scan diffs for secrets automatically?
• ☐ Have I documented the finding for the team?

FAQ

Is AI-generated code more likely to contain secrets than human-written code?

Yes, for two reasons. First, AI agents generate realistic-looking credentials from training data patterns. Second, agents have no concept of „sensitive” — they treat API keys and passwords the same as variable names. Human developers at least have an instinct to avoid hardcoding secrets, even if they sometimes fail.

Can I rely on GitHub’s built-in secret scanning?

GitHub’s secret scanning is a good safety net, but it has limitations: it only scans public repositories on free plans, it has a fixed set of patterns (no custom patterns without Advanced Security), and it does not catch secrets in private repos on Team plans. It also does not scan diffs specifically — it scans the full repository state. For AI code review, a diff-focused scanner catches changes earlier and more precisely.

What about pre-commit hooks?

Pre-commit hooks are excellent for catching secrets before they enter git history. Tools like gitleaks and detect-secrets have pre-commit integrations. The limitation: pre-commit hooks only catch what you stage — they do not review the AI agent’s full change scope. Use both pre-commit hooks and diff-focused scanning for full coverage.

Should I scan for secrets in test fixtures?

Yes. AI-generated test fixtures frequently contain realistic-looking secrets that follow real format patterns. Even if they are not real credentials, they trigger false positives in automated scanners and can confuse other developers. Replace them with clearly fake values or environment variable references.

How do I handle secrets that the agent generates and then removes?

If a secret appears in any commit in your branch’s history, treat it as compromised. The agent may have „refactored” the secret into an environment variable, but the original value is still in the diff and git history. Rotate the secret and scan the full branch history.

What is the difference between full-repo scanning and diff scanning?

Full-repo scanning checks all files in your repository for secrets. Diff scanning checks only what changed in a specific commit or pull request. For AI code review, diff scanning is more precise and faster — it focuses on exactly what the agent changed and ignores unchanged files that you have already reviewed.

Bottom line

Secret leakage is one of the most concrete, measurable risks in AI-generated code. Unlike vague „code quality” concerns, a leaked API key has immediate, quantifiable impact. Every AI-assisted pull request should be scanned for secrets before merge — not as a one-time check, but as a systematic part of your review workflow.

Start with a diff-focused scanner, add CI integration, and build the habit of reviewing config file changes with extra scrutiny. The checklist above takes under 5 minutes per PR and catches the most common patterns.

CodeRiskTools Scanner 3.0.0 is the current public scanner: MIT-licensed and free on GitHub for local secret and configuration-change review.

Related articles

• How to review AI-generated code before you merge it
• CI gates for AI-generated code: stop risky changes before they reach production
• AI code review checklist for small software teams
• Agentic coding risk review: a practical workflow for teams using AI coding agents

Browse all CodeRiskTools topics

Expert Audit intake is paused. CodeRiskTools is not accepting new private code audit orders, and no turnaround is promised.

How to detect secrets in code before merge

Google Search Console is now showing CodeRiskTools for detect secrets in code and for this secret-scanning article. The practical workflow is simple: scan the diff before merge, review generated configuration changes, and block credentials before they reach the default branch.

What counts as a secret in AI-generated code?

Secrets include API keys, OAuth tokens, webhooks, database passwords, private keys, cloud credentials, session cookies, and realistic placeholder values that could be copied into production. AI coding tools can also create risky examples in README files, test fixtures, Docker Compose files, environment samples, and deployment scripts.

Local secret scanning checklist

• Scan changed files before commit, not only after CI runs.
• Review .env, YAML, JSON, Docker, GitHub Actions, and deployment files manually.
• Block long token-looking strings, private key headers, and credential assignments.
• Check generated documentation and tests, because AI often places fake-looking secrets there.
• Use a config diff review when the AI assistant changed defaults, endpoints, permissions, or environment names.

When to use a config Scanner 3.0.0

Use a config Scanner 3.0.0 when AI changes touch deployment, CI, environment variables, infrastructure files, package scripts, or auth-sensitive settings. Secret scanning catches exposed values; config diff review catches dangerous configuration drift around those values.

Use Scanner 3.0.0 on GitHub when you need MIT/free local review of secrets and configuration changes.

How to detect secrets in code before merge

Short answer: scan the diff, generated files, configuration changes, tests, fixtures, logs, and CI output before a merge. AI agents can accidentally copy tokens, example credentials, private keys, webhook URLs, or cloud identifiers into places reviewers do not normally inspect.

What counts as a secret in AI-generated code?

• API keys, bearer tokens, OAuth secrets, webhook signing secrets, and service-account files.
• Private keys, seed material, certificates, SSH keys, and cloud credential JSON.
• Database URLs, password-bearing connection strings, `.env` dumps, and copied CI variables.
• Internal hostnames, privileged endpoints, temporary debug credentials, and production-like fixtures.

Local secret scanning checklist

Run a secret scanner on the staged diff, inspect generated tests and fixtures manually, check any new `.env.example` or config file, and rotate immediately if a real credential was committed. Treat AI agent secret leakage as a release blocker until the credential is removed, history exposure is assessed, and rotation evidence is recorded.

Related: CI gates for AI-generated code, AI code review before merge, and CodeRiskTools products.