Meta

• License: Apache Software License
• Author: Yelp, Inc.
• Tags secret-management , pre-commit , security , entropy-checks

Classifiers

• Development Status
  • 5 - Production/Stable
• Environment
  • Console
• Intended Audience
  • Developers
• License
  • OSI Approved :: Apache Software License
• Operating System
  • OS Independent
• Programming Language
  • Python :: 3
• Topic
  • Software Development
  • Utilities
• Typing
  • Typed

👁 Build Status
👁 PyPI version
👁 Homebrew
👁 PRs Welcome
👁 AMF

detect-secrets

About

detect-secrets is an aptly named module for (surprise, surprise) detecting secrets within a code base.

However, unlike other similar packages that solely focus on finding secrets, this package is designed with the enterprise client in mind: providing a backwards compatible, systematic means of:

1. Preventing new secrets from entering the code base,
2. Detecting if such preventions are explicitly bypassed, and
3. Providing a checklist of secrets to roll, and migrate off to a more secure storage.

This way, you create a separation of concern: accepting that there may currently be secrets hiding in your large repository (this is what we refer to as a baseline), but preventing this issue from getting any larger, without dealing with the potentially gargantuan effort of moving existing secrets away.

It does this by running periodic diff outputs against heuristically crafted regex statements, to identify whether any new secret has been committed. This way, it avoids the overhead of digging through all git history, as well as the need to scan the entire repository every time.

For a look at recent changes, please see CHANGELOG.md.

If you are looking to contribute, please see CONTRIBUTING.md.

For more detailed documentation, check out our other documentation.

Examples

Quickstart:

Create a baseline of potential secrets currently found in your git repository.

$detect-secretsscan>.secrets.baseline

or, to run it from a different directory:

$detect-secrets-C/path/to/directoryscan>/path/to/directory/.secrets.baseline

Scanning non-git tracked files:

$detect-secretsscantest_data/--all-files>.secrets.baseline

Adding New Secrets to Baseline:

This will rescan your codebase, and:

1. Update/upgrade your baseline to be compatible with the latest version,
2. Add any new secrets it finds to your baseline,
3. Remove any secrets no longer in your codebase

This will also preserve any labelled secrets you have.

$detect-secretsscan--baseline.secrets.baseline

For baselines older than version 0.9, just recreate it.

Alerting off newly added secrets:

Scanning Staged Files Only:

$gitdiff--staged--name-only-z|xargs-0detect-secrets-hook--baseline.secrets.baseline

Scanning All Tracked Files:

$gitls-files-z|xargs-0detect-secrets-hook--baseline.secrets.baseline

Viewing All Enabled Plugins:

$detect-secretsscan--list-all-plugins
ArtifactoryDetector
AWSKeyDetector
AzureStorageKeyDetector
BasicAuthDetector
CloudantDetector
DiscordBotTokenDetector
GitHubTokenDetector
GitLabTokenDetector
Base64HighEntropyString
HexHighEntropyString
IbmCloudIamDetector
IbmCosHmacDetector
IPPublicDetector
JwtTokenDetector
KeywordDetector
MailchimpDetector
NpmDetector
OpenAIDetector
PrivateKeyDetector
PypiTokenDetector
SendGridDetector
SlackDetector
SoftlayerDetector
SquareOAuthDetector
StripeDetector
TelegramBotTokenDetector
TwilioKeyDetector

Disabling Plugins:

$detect-secretsscan--disable-pluginKeywordDetector--disable-pluginAWSKeyDetector

If you want to only run a specific plugin, you can do:

$detect-secretsscan--list-all-plugins|\
grep-v'BasicAuthDetector'|\
sed"s#^#--disable-plugin #g"|\
xargsdetect-secretsscantest_data

Auditing a Baseline:

This is an optional step to label the results in your baseline. It can be used to narrow down your checklist of secrets to migrate, or to better configure your plugins to improve its signal-to-noise ratio.

$detect-secretsaudit.secrets.baseline

Usage in Other Python Scripts

Basic Use:

from detect_secrets import SecretsCollection
from detect_secrets.settings import default_settings

secrets = SecretsCollection()
with default_settings():
 secrets.scan_file('test_data/config.ini')


import json
print(json.dumps(secrets.json(), indent=2))

More Advanced Configuration:

from detect_secrets import SecretsCollection
from detect_secrets.settings import transient_settings

secrets = SecretsCollection()
with transient_settings({
 # Only run scans with only these plugins.
 # This format is the same as the one that is saved in the generated baseline.
 'plugins_used': [
 # Example of configuring a built-in plugin
 {
 'name': 'Base64HighEntropyString',
 'limit': 5.0,
 },

 # Example of using a custom plugin
 {
 'name': 'HippoDetector',
 'path': 'file:///Users/aaronloo/Documents/github/detect-secrets/testing/plugins.py',
 },
 ],

 # We can also specify whichever additional filters we want.
 # This is an example of using the function `is_identified_by_ML_model` within the
 # local file `./private-filters/example.py`.
 'filters_used': [
 {
 'path': 'file://private-filters/example.py::is_identified_by_ML_model',
 },
 ]
}) as settings:
 # If we want to make any further adjustments to the created settings object (e.g.
 # disabling default filters), we can do so as such.
 settings.disable_filters(
 'detect_secrets.filters.heuristic.is_prefixed_with_dollar_sign',
 'detect_secrets.filters.heuristic.is_likely_id_string',
 )

 secrets.scan_file('test_data/config.ini')

Installation

$pipinstalldetect-secrets
✨🍰✨

Install via brew:

$brewinstalldetect-secrets

Usage

detect-secrets comes with three different tools, and there is often confusion around which one to use. Use this handy checklist to help you decide:

1. Do you want to add secrets to your baseline? If so, use detect-secrets scan.
2. Do you want to alert off new secrets not in the baseline? If so, use detect-secrets-hook.
3. Are you analyzing the baseline itself? If so, use detect-secrets audit.

Adding Secrets to Baseline

$ detect-secrets scan --help
usage: detect-secrets scan [-h] [--string [STRING]] [--only-allowlisted]
 [--all-files] [--baseline FILENAME]
 [--force-use-all-plugins] [--slim]
 [--list-all-plugins] [-p PLUGIN]
 [--base64-limit [BASE64_LIMIT]]
 [--hex-limit [HEX_LIMIT]]
 [--disable-plugin DISABLE_PLUGIN]
 [-n | --only-verified]
 [--exclude-lines EXCLUDE_LINES]
 [--exclude-files EXCLUDE_FILES]
 [--exclude-secrets EXCLUDE_SECRETS]
 [--word-list WORD_LIST_FILE] [-f FILTER]
 [--disable-filter DISABLE_FILTER]
 [path [path ...]]

Scans a repository for secrets in code. The generated output is compatible
with `detect-secrets-hook --baseline`.

positional arguments:
 path Scans the entire codebase and outputs a snapshot of
 currently identified secrets.

optional arguments:
 -h, --help show this help message and exit
 --string [STRING] Scans an individual string, and displays configured
 plugins' verdict.
 --only-allowlisted Only scans the lines that are flagged with `allowlist
 secret`. This helps verify that individual exceptions
 are indeed non-secrets.

scan options:
 --all-files Scan all files recursively (as compared to only
 scanning git tracked files).
 --baseline FILENAME If provided, will update existing baseline by
 importing settings from it.
 --force-use-all-plugins
 If a baseline is provided, detect-secrets will default
 to loading the plugins specified by that baseline.
 However, this may also mean it doesn't perform the
 scan with the latest plugins. If this flag is
 provided, it will always use the latest plugins
 --slim Slim baselines are created with the intention of
 minimizing differences between commits. However, they
 are not compatible with the `audit` functionality, and
 slim baselines will need to be remade to be audited.

plugin options:
 Configure settings for each secret scanning ruleset. By default, all
 plugins are enabled unless explicitly disabled.

 --list-all-plugins Lists all plugins that will be used for the scan.
 -p PLUGIN, --plugin PLUGIN
 Specify path to custom secret detector plugin.
 --base64-limit [BASE64_LIMIT]
 Sets the entropy limit for high entropy strings. Value
 must be between 0.0 and 8.0, defaults to 4.5.
 --hex-limit [HEX_LIMIT]
 Sets the entropy limit for high entropy strings. Value
 must be between 0.0 and 8.0, defaults to 3.0.
 --disable-plugin DISABLE_PLUGIN
 Plugin class names to disable. e.g.
 Base64HighEntropyString

filter options:
 Configure settings for filtering out secrets after they are flagged by the
 engine.

 -n, --no-verify Disables additional verification of secrets via
 network call.
 --only-verified Only flags secrets that can be verified.
 --exclude-lines EXCLUDE_LINES
 If lines match this regex, it will be ignored.
 --exclude-files EXCLUDE_FILES
 If filenames match this regex, it will be ignored.
 --exclude-secrets EXCLUDE_SECRETS
 If secrets match this regex, it will be ignored.
 --word-list WORD_LIST_FILE
 Text file with a list of words, if a secret contains a
 word in the list we ignore it.
 -f FILTER, --filter FILTER
 Specify path to custom filter. May be a python module
 path (e.g.
 detect_secrets.filters.common.is_invalid_file) or a
 local file path (e.g.
 file://path/to/file.py::function_name).
 --disable-filter DISABLE_FILTER
 Specify filter to disable. e.g.
 detect_secrets.filters.common.is_invalid_file

Blocking Secrets not in Baseline

$ detect-secrets-hook --help
usage: detect-secrets-hook [-h] [-v] [--version] [--baseline FILENAME]
 [--list-all-plugins] [-p PLUGIN]
 [--base64-limit [BASE64_LIMIT]]
 [--hex-limit [HEX_LIMIT]]
 [--disable-plugin DISABLE_PLUGIN]
 [-n | --only-verified]
 [--exclude-lines EXCLUDE_LINES]
 [--exclude-files EXCLUDE_FILES]
 [--exclude-secrets EXCLUDE_SECRETS]
 [--word-list WORD_LIST_FILE] [-f FILTER]
 [--disable-filter DISABLE_FILTER]
 [filenames [filenames ...]]

positional arguments:
 filenames Filenames to check.

optional arguments:
 -h, --help show this help message and exit
 -v, --verbose Verbose mode.
 --version Display version information.
 --json Print detect-secrets-hook output as JSON
 --baseline FILENAME Explicitly ignore secrets through a baseline generated
 by `detect-secrets scan`

plugin options:
 Configure settings for each secret scanning ruleset. By default, all
 plugins are enabled unless explicitly disabled.

 --list-all-plugins Lists all plugins that will be used for the scan.
 -p PLUGIN, --plugin PLUGIN
 Specify path to custom secret detector plugin.
 --base64-limit [BASE64_LIMIT]
 Sets the entropy limit for high entropy strings. Value
 must be between 0.0 and 8.0, defaults to 4.5.
 --hex-limit [HEX_LIMIT]
 Sets the entropy limit for high entropy strings. Value
 must be between 0.0 and 8.0, defaults to 3.0.
 --disable-plugin DISABLE_PLUGIN
 Plugin class names to disable. e.g.
 Base64HighEntropyString

filter options:
 Configure settings for filtering out secrets after they are flagged by the
 engine.

 -n, --no-verify Disables additional verification of secrets via
 network call.
 --only-verified Only flags secrets that can be verified.
 --exclude-lines EXCLUDE_LINES
 If lines match this regex, it will be ignored.
 --exclude-files EXCLUDE_FILES
 If filenames match this regex, it will be ignored.
 --exclude-secrets EXCLUDE_SECRETS
 If secrets match this regex, it will be ignored.
 -f FILTER, --filter FILTER
 Specify path to custom filter. May be a python module
 path (e.g.
 detect_secrets.filters.common.is_invalid_file) or a
 local file path (e.g.
 file://path/to/file.py::function_name).
 --disable-filter DISABLE_FILTER
 Specify filter to disable. e.g.
 detect_secrets.filters.common.is_invalid_file

We recommend setting this up as a pre-commit hook. One way to do this is by using the pre-commit framework:

# .pre-commit-config.yaml
repos:
-repo:https://github.com/Yelp/detect-secrets
rev:v1.5.0
hooks:
-id:detect-secrets
args:['--baseline','.secrets.baseline']
exclude:package.lock.json

Inline Allowlisting

There are times when we want to exclude a false positive from blocking a commit, without creating a baseline to do so. You can do so by adding a comment as such:

secret = "hunter2" # pragma: allowlist secret

or

// pragma: allowlist nextline secret
constsecret="hunter2";

Auditing Secrets in Baseline

$detect-secretsaudit--help
usage:detect-secretsaudit[-h][--diff][--stats]
[--report][--only-real|--only-false]
[--json]
filename[filename...]

Auditingabaselineallowsanalyststolabelresults,andoptimizepluginsfor
thehighestsignal-to-noiseratiofortheirenvironment.

positionalarguments:
filenameAuditagivenbaselinefiletodistinguishthedifference
betweenfalseandtruepositives.

optionalarguments:
-h,--helpshowthishelpmessageandexit
--diffAllowsthecomparisonoftwobaselinefiles,inorderto
effectivelydistinguishthedifferencebetweenvariousplugin
configurations.
--statsDisplaystheresultsofaninteractiveauditingsessionwhich
havebeensavedtoabaselinefile.
--reportDisplaysareportwiththesecretsdetected

reporting:
Displayasummarywithallthefindingsandthemadedecisions.Tobeusedwiththereportmode(--report).

--only-realOnlyincludesrealsecretsinthereport
--only-falseOnlyincludesfalsepositivesinthereport

analytics:
Quantifythesuccessofyourpluginsbasedonthelabelledresultsinyour
baseline.Tobeusedwiththestatisticsmode(--stats).

--jsonOutputsresultsinamachine-readableformat.

Configuration

This tool operates through a system of plugins and filters.

• Plugins find secrets in code
• Filters ignore false positives to increase scanning precision

You can adjust both to suit your precision/recall needs.

Plugins

There are three different strategies we employ to try and find secrets in code:

1. Regex-based Rules

   These are the most common type of plugin, and work well with well-structured secrets. These secrets can optionally be verified, which increases scanning precision. However, solely depending on these may negatively affect the recall of your scan.

2. Entropy Detector

   This searches for "secret-looking" strings through a variety of heuristic approaches. This is great for non-structured secrets, but may require tuning to adjust the scanning precision.

3. Keyword Detector

   This ignores the secret value, and searches for variable names that are often associated with assigning secrets with hard-coded values. This is great for "non-secret-looking" strings (e.g. le3tc0de passwords), but may require tuning filters to adjust the scanning precision.

Want to find a secret that we don't currently catch? You can also (easily) develop your own plugin, and use it with the engine! For more information, check out the plugin documentation.

Filters

detect-secrets comes with several different in-built filters that may suit your needs.

--exclude-lines

Sometimes, you want to be able to globally allow certain lines in your scan, if they match a specific pattern. You can specify a regex rule as such:

$detect-secretsscan--exclude-lines'password = (blah|fake)'

Or you can specify multiple regex rules as such:

$detect-secretsscan--exclude-lines'password = blah'--exclude-lines'password = fake'

--exclude-files

Sometimes, you want to be able to ignore certain files in your scan. You can specify a regex pattern to do so, and if the filename meets this regex pattern, it will not be scanned:

$detect-secretsscan--exclude-files'.*\.signature$'

Or you can specify multiple regex patterns as such:

$detect-secretsscan--exclude-files'.*\.signature$'--exclude-files'.*/i18n/.*'

--exclude-secrets

Sometimes, you want to be able to ignore certain secret values in your scan. You can specify a regex rule as such:

$detect-secretsscan--exclude-secrets'(fakesecret|\${.*})'

Or you can specify multiple regex rules as such:

$detect-secretsscan--exclude-secrets'fakesecret'--exclude-secrets'\${.*})'

Inline Allowlisting

Sometimes, you want to apply an exclusion to a specific line, rather than globally excluding it. You can do so with inline allowlisting as such:

API_KEY = 'this-will-ordinarily-be-detected-by-a-plugin' # pragma: allowlist secret

These comments are supported in multiple languages. e.g.

constGoogleCredentialPassword="something-secret-here";// pragma: allowlist secret

You can also use:

# pragma: allowlist nextline secret
API_KEY = 'WillAlsoBeIgnored'

This may be a convenient way for you to ignore secrets, without needing to regenerate the entire baseline again. If you need to explicitly search for these allowlisted secrets, you can also do:

$detect-secretsscan--only-allowlisted

Want to write more custom logic to filter out false positives? Check out how to do this in our filters documentation.

Extensions

wordlist

The --exclude-secrets flag allows you to specify regex rules to exclude secret values. However, if you want to specify a large list of words instead, you can use the --word-list flag.

To use this feature, be sure to install the pyahocorasick package, or simply use:

$pipinstalldetect-secrets[word_list]

Then, you can use it as such:

$catwordlist.txt
not-a-real-secret
$catsample.ini
password=not-a-real-secret

# Will show results
$detect-secretsscansample.ini

# No results found
$detect-secretsscan--word-listwordlist.txt

Gibberish Detector

The Gibberish Detector is a simple ML model, that attempts to determine whether a secret value is actually gibberish, with the assumption that real secret values are not word-like.

To use this feature, be sure to install the gibberish-detector package, or use:

$pipinstalldetect-secrets[gibberish]

Check out the gibberish-detector package for more information on how to train the model. A pre-trained model (seeded by processing RFCs) will be included for easy use.

You can also specify your own model as such:

$detect-secretsscan--gibberish-modelcustom.model

This is not a default plugin, given that this will ignore secrets such as password.

Caveats

This is not meant to be a sure-fire solution to prevent secrets from entering the codebase. Only proper developer education can truly do that. This pre-commit hook merely implements several heuristics to try and prevent obvious cases of committing secrets.

Things That Won't Be Prevented:

• Multi-line secrets
• Default passwords that don't trigger the KeywordDetector (e.g. login = "hunter2")

FAQ

General

• "Did not detect git repository." warning encountered, even though I'm in a git repo.

  Check to see whether your git version is >= 1.8.5. If not, please upgrade it then try again. More details here.

Windows

• detect-secrets audit displays "Not a valid baseline file!" after creating baseline.

  Ensure the file encoding of your baseline file is UTF-8. More details here.

Verified details

These details have been verified by PyPI

Maintainers

👁 Avatar for domanchi from gravatar.com
domanchi 👁 Avatar for KevinHock from gravatar.com
KevinHock 👁 Avatar for yelplabs from gravatar.com
yelplabs

Unverified details

These details have not been verified by PyPI

Project links

• Homepage
• Download

• Download URL: detect_secrets-1.5.0.tar.gz
• Upload date: May 6, 2024
• Size: 97.4 kB
• Tags: Source
• Uploaded using Trusted Publishing? No
• Uploaded via: twine/5.0.0 CPython/3.9.19

Hashes for detect_secrets-1.5.0.tar.gz

Algorithm	Hash digest
SHA256	6bb46dcc553c10df51475641bb30fd69d25645cc12339e46c824c1e0c388898a
MD5	c21d09fc490c1316b4aa4d066a108b24
BLAKE2b-256	6967382a863fff94eae5a0cf05542179169a1c49a4c8784a9480621e2066ca7d

See more details on using hashes here.

• Download URL: detect_secrets-1.5.0-py3-none-any.whl
• Upload date: May 6, 2024
• Size: 120.3 kB
• Tags: Python 3
• Uploaded using Trusted Publishing? No
• Uploaded via: twine/5.0.0 CPython/3.9.19

Hashes for detect_secrets-1.5.0-py3-none-any.whl

Algorithm	Hash digest
SHA256	e24e7b9b5a35048c313e983f76c4bd09dad89f045ff059e354f9943bf45aa060
MD5	803b3510c603c29fec9ed333346e0e0c
BLAKE2b-256	4e5e4f5fe4b89fde1dc3ed0eb51bd4ce4c0bca406246673d370ea2ad0c58d747

See more details on using hashes here.