Pre-receive secret detection troubleshooting

When to use this runbook?

Use this runbook for troubleshooting Production issues related to the pre-receive secret detection feature.

Relevant settings

Setting	Type	Level	Visibility
pre_receive_secret_detection_beta_release	Feature Flag	Instance	Not visible, has to be toggled via ChatOps.
pre_receive_secret_detection_enabled	Database Setting	Instance	Only in a Dedicated instance, or with pre_receive_secret_detection_beta_release enabled and only when the feature is licensed (in Ultimate).
pre_receive_secret_detection_push_check	Feature Flag	Project	Not visible, has to be toggled via ChatOps.
pre_receive_secret_detection_enabled	Database Setting	Project	Only in a Dedicated instance, or with pre_receive_secret_detection_enabled enabled and only when the feature is licensed (in Ultimate).

Monitoring

[DRAFT] Pre-receive secret detection monitoring is the preferred dashboard for monitoring the feature to help identify and mitigate any reliability issues or performance regressions that may occur when it is enabled on Gitlab.com.

Gitaly Latency Dashboard for the PreReceiveHook method may be used as a supplement.

Enabling/Disabling

Enabling/Disabling for Gitlab.com

To disable the feature across all of Gitlab.com, disable pre-receive secret detection, as described in the docs. This will require a change access request (production example), as it requires administrator access.

This will disable the feature for all Gitlab.com users and should be used in cases where the feature is causing significant performance degradition.

Other properties that can be enabled/disabled

Enabling/Disabling for a project

This can only be changed by a Maintainer, or above, for the project and is the preferred method for disabling a project.

With the pre_receive_secret_detection_push_check feature flag on for a given project, pre-receive secret detection can be toggled for the project through the security configuration page for the project. Instructions can be found in this in-progress documentation MR, but will eventually be added to the docs.

When both the feature flag and the setting are on, git pushes to the project should be stopped when they contain a secret. As a test, glpat-12345678901234567890 should be prevented when pushing (from the terminal, the Web IDE, etc.)

The pre_receive_secret_detection_push_check feature flag

The pre_receive_secret_detection_push_check feature flag will be used to enable/disable pre-receive secret detection for a given project.

It will also be used in conjunction with the pre-receive secret detection setting in a project’s Security Configuration. Both the feature flag and the setting will need to be on for pre-receive secret detection to work for a given project.

Check status:

/chatops run feature get pre_receive_secret_detection_push_check

Enable for a project:

/chatops run feature set --project=the-namespace/of-the-project pre_receive_secret_detection_push_check true

Disable for a project:

/chatops run feature set --project=the-namespace/of-the-project pre_receive_secret_detection_push_check false

The pre_receive_secret_detection_beta_release flag

The pre_receive_secret_detection_beta_release flag will be used to enable the Secret Detection section in the Security and Compliance Admin section (https://gitlab.com/admin -> Settings -> Security and Compliance) , but turning the flag on does not turn on pre-receive secret detection. Conversely, turning the flag off does not turn the feature off.

Check the status of the pre_receive_secret_detection_beta_release feature flag:

/chatops run feature get pre_receive_secret_detection_beta_release

Enable the feature flag:

/chatops run feature set pre_receive_secret_detection_beta_release true

Disable the feature flag:

/chatops run feature set pre_receive_secret_detection_beta_release false

Resolving and skipping blocked pushes

Resolving a blocked push and Skipping secret detection are both documented in the in-progress documentation MR.