Sign commits and tags with X.509 certificates

DETAILS: Tier: Free, Premium, Ultimate Offering: Self-managed, GitLab Dedicated

X.509 is a standard format for public key certificates issued by a public or private Public Key Infrastructure (PKI). Personal X.509 certificates are used for authentication or signing purposes such as S/MIME (Secure/Multipurpose Internet Mail Extensions). However, Git also supports signing of commits and tags with X.509 certificates in a similar way as with GPG (GnuPG, or GNU Privacy Guard). The main difference is the way GitLab determines whether or not the developer's signature is trusted:

For X.509, a root certificate authority is added to the GitLab trust store. (A trust store is a repository of trusted security certificates.) Combined with any required intermediate certificates in the signature, the developer's certificate can be chained back to a trusted root certificate.
For GPG, developers add their GPG key to their account.

GitLab uses its own certificate store and therefore defines the trust chain. For a commit or tag to be verified by GitLab:

The signing certificate email must match a verified email address in GitLab.
The GitLab instance must be able to establish a full trust chain from the certificate in the signature to a trusted certificate in the GitLab certificate store. This chain may include intermediate certificates supplied in the signature. You may need to add certificates, such as Certificate Authority root certificates, to the GitLab certificate store.
The signing time must be in the time range of the certificate validity, which is usually up to three years.
The signing time is equal to, or later than, the commit time.

If a commit's status has already been determined and stored in the database, use the Rake task to re-check the status. Refer to the Troubleshooting section. GitLab checks certificate revocation lists on a daily basis with a background worker.

Known issues

Certificates without authorityKeyIdentifier, subjectKeyIdentifier, and crlDistributionPoints display as Unverified. We recommend using certificates from a PKI that are in line with RFC 5280.
The Verified badge is not displayed on the GitLab.com offering because uploading a custom Certification Authority (CA) is only available to self-managed instances.
Setting values in the Extended Key Usage (EKU) section of your certificate in addition to the required Key Usage (KU) of Digital Signature is likely to cause your commits to display as Unverified. To resolve this, add emailProtection to your EKU list. RFC 5280 specifies this limitation.

To diagnose it, follow S/MIME verification with OpenSSL. If this change doesn't resolve the issue, please provide feedback in issue 440189.
In GitLab 16.2 and earlier, if you have more than one email in the Subject Alternative Name list in your signing certificate, only the first one is used to verify commits.
In GitLab 15.1 and earlier, the X509v3 Subject Key Identifier (SKI) in the issuer certificate and the signing certificate must be 40 characters long. If your SKI is shorter, commits don't show as verified in GitLab, and short subject key identifiers may also cause errors when accessing the project, such as 'An error occurred while loading commit signatures' and HTTP 422 Unprocessable Entity errors.

Configure for signed commits

To sign your commits, tags, or both, you must:

Obtain an X.509 key pair.
Associate your X.509 certificate with Git.
Sign and verify commits.
Sign and verify tags.

Obtain an X.509 key pair

If your organization has Public Key Infrastructure (PKI), that PKI provides an S/MIME key. If you do not have an S/MIME key pair from a PKI, you can either create your own self-signed pair, or purchase a pair.

Associate your X.509 certificate with Git

To take advantage of X.509 signing, you need Git 2.19.0 or later. You can check your Git version with the command git --version.

If you have the correct version, you can proceed to configure Git.

Linux

Configure Git to use your key for signing:

signingkey=$( gpgsm --list-secret-keys | egrep '(key usage|ID)' | grep -B 1 digitalSignature | awk '/ID/ {print $2}' )
git config --global user.signingkey $signingkey
git config --global gpg.format x509

Windows and macOS

To configure Windows or macOS:

Install S/MIME Sign by either:
- Downloading the installer.
- Running brew install smimesign on macOS.
Get the ID of your certificate by running smimesign --list-keys.
Set your signing key by running git config --global user.signingkey <ID>, replacing <ID> with the certificate ID.

Configure X.509 with this command:

git config --global gpg.x509.program smimesign
git config --global gpg.format x509

Sign and verify commits

After you have associated your X.509 certificate with Git you can sign your commits:

When you create a Git commit, add the -S flag:

git commit -S -m "feat: x509 signed commits"

Push to GitLab, and check that your commits are verified with the --show-signature flag:
```
git log --show-signature
```
If you don't want to type the -S flag every time you commit, run this command for Git to sign your commits every time:
```
git config --global commit.gpgsign true
```

Sign and verify tags

After you have associated your X.509 certificate with Git you can start signing your tags:

When you create a Git tag, add the -s flag:
```
git tag -s v1.1.1 -m "My signed tag"
```
Push to GitLab and verify your tags are signed with this command:
```
git tag --verify v1.1.1
```
If you don't want to type the -s flag every time you tag, run this command for Git to sign your tags each time:
```
git config --global tag.gpgsign true
```

Troubleshooting

For committers without administrator access, review the list of verification problems with signed commits for possible fixes. The other troubleshooting suggestions on this page require administrator access.

Re-verify commits

GitLab stores the status of any checked commits in the database. You can use a Rake task to check the status of any previously checked commits.

After you make any changes, run this command:

sudo gitlab-rake gitlab:x509:update_signatures

Main verification checks

The code performs these key checks, which all must return verified:

x509_certificate.nil? should be false.
x509_certificate.revoked? should be false.
verified_signature should be true.
user.nil? should be false.
user.verified_emails.include?(@email) should be true.
certificate_email == @email should be true.

To investigate why a commit shows as Unverified:

Start a Rails console:
```
sudo gitlab-rails console
```

Identify the project (either by path or ID) and full commit SHA that you're investigating. Use this information to create signature to run other checks against:

project = Project.find_by_full_path('group/subgroup/project')
project = Project.find_by_id('121')
commit = project.repository.commit_by(oid: '87fdbd0f9382781442053b0b76da729344e37653')
signedcommit=Gitlab::X509::Commit.new(commit)
signature=Gitlab::X509::Signature.new(signedcommit.signature_text, signedcommit.signed_text, commit.committer_email, commit.created_at)

If you make changes to address issues identified running through the checks, restart the Rails console and run though the checks again from the start.

Check the certificate on the commit:

signature.x509_certificate.nil?
signature.x509_certificate.revoked?

Both checks should return false:

> signature.x509_certificate.nil?
=> false
> signature.x509_certificate.revoked?
=> false

A known issue causes these checks to fail with Validation failed: Subject key identifier is invalid.

Run a cryptographic check on the signature. The code must return true:
```
signature.verified_signature
```
If it returns false then investigate this check further.
Confirm the email addresses match on the commit and the signature:
- The Rails console displays the email addresses being compared.
- The final command must return true:
```
sigemail=signature.__send__:certificate_email
commitemail=commit.committer_email
sigemail == commitemail
```
In GitLab 16.2 and earlier, only the first email in the Subject Alternative Name list is compared. To display the Subject Alternative Name list, run:
```
signature.__send__ :get_certificate_extension,'subjectAltName'
```
If the developer's email address is not the first one in the list, this check fails, and the commit is marked unverified.
The email address on the commit must be associated with an account in GitLab. This check should return false:
```
signature.user.nil?
```
Check the email address is associated with a user in GitLab. This check should return a user, such as #<User id:1234 @user_handle>:
```
User.find_by_any_email(commit.committer_email)
```
If it returns nil, the email address is not associated with a user, and the check fails.
Confirm the developer's email address is verified. This check must return true:
```
signature.user.verified_emails.include?(commit.committer_email)
```
If the previous check returned nil, this command displays an error:
```
NoMethodError (undefined method `verified_emails' for nil:NilClass)
```
The verification status is stored in the database. To display the database record:
```
pp CommitSignatures::X509CommitSignature.by_commit_sha(commit.sha);nil
```
If all the previous checks returned the correct values:
- verification_status: "unverified" indicates the database record needs updating. Use the Rake task.
- [] indicates the database doesn't have a record yet. Locate the commit in GitLab to check the signature and store the result.

Cryptographic verification checks

If GitLab determines that verified_signature is false, investigate the reason in the Rails console. These checks require signature to exist. Refer to the signature step of the previous main verification checks.

Check the signature, without checking the issuer, returns true:
```
signature.__send__ :valid_signature?
```
Check the signing time and date. This check must return true:
```
signature.__send__ :valid_signing_time?
```
- The code allows for code signing certificates to expire.
- A commit must be signed during the validity period of the certificate, and at or after the commit's datestamp. Display the commit time and certificate details including not_before, not_after with:
```
commit.created_at
pp signature.__send__ :cert; nil
```
Check the signature, including that TLS trust can be established. This check must return true:
```
signature.__send__(:p7).verify([], signature.__send__(:cert_store), signature.__send__(:signed_text))
```
1. If this fails, add the missing certificates required to establish trust to the GitLab certificate store.
2. After adding more certificates, (if these troubleshooting steps then pass) run the Rake task to re-verify commits.
3. You can add additional certificates dynamically in the Rails console to check if this resolves the problem.
  1. Retest the signature with a trust store cert_store that can be modified. It should still fail, with false:
```
cert_store = signature.__send__ :cert_store
signature.__send__(:p7).verify([], cert_store, signature.__send__(:signed_text))
```
  2. Add an additional certificate, and re-test:
```
cert_store.add_file("/etc/ssl/certs/my_new_root_ca.pem")
signature.__send__(:p7).verify([], cert_store, signature.__send__(:signed_text))
```
4. Display the certificates that are included in the signature:
```
pp signature.__send__(:p7).certificates ; nil
```
5. Further investigation can be performed with OpenSSL on the command line.

Ensure any additional intermediate certificates and the root certificate are added to the certificate store. For consistency with how certificate chains are built on web servers:

Git clients that are signing commits should include the certificate and all intermediate certificates in the signature.
The GitLab certificate store should only contain the root.

If you remove a root certificate from the GitLab trust store, such as when it expires, commit signatures which chain back to that root display as unverified.

S/MIME verification with OpenSSL

If there are issues with the signature, or if TLS trust fails, further debugging can be performed with OpenSSL on the command line.

Export the signature and the signed text, from the Rails console:

The initial two steps from the main verification checks are required so signature has been set.

OpenSSL requires that PKCS7 PEM formatted data is bounded with BEGIN PKCS7 and END PKCS7 so this usually needs to be fixed:

pkcs7_text = signature.signature_text.sub('-----BEGIN SIGNED MESSAGE-----', '-----BEGIN PKCS7-----')
pkcs7_text = pkcs7_text.sub('-----END SIGNED MESSAGE-----', '-----END PKCS7-----')

Write out the signature and signed text:

f1=File.new('/tmp/signature_text.pk7.pem','w')
f1 << pkcs7_text
f1.close

f2=File.new('/tmp/signed_text.txt','w')
f2 << signature.signed_text
f2.close

This data can now be investigated on the Linux command line using OpenSSL:

The PKCS #7 file containing the signature can be queried:

/opt/gitlab/embedded/bin/openssl pkcs7 -inform pem -print_certs \
    -in /tmp/signature_text.pk7.pem -print -noout

It should include at least one cert section in the output; the signer's certificate.

There's a lot of low level of detail in the output. Here's an example of some of the structure and headings that should be present:

PKCS7:
  d.sign:
    cert:
        cert_info:
          issuer:
          validity:
            notBefore:
            notAfter:
          subject:

If developers' code signing certificates are issued by an intermediate certificate authority, there should be additional certificate details:

PKCS7:
  d.sign:
    cert:
        cert_info:
    cert:
        cert_info:

Extract the certificate from the signature:
```
/opt/gitlab/embedded/bin/openssl pkcs7 -inform pem -print_certs \
    -in /tmp/signature_text.pk7.pem -out /tmp/signature_cert.pem
```
If this step fails, the signature might be missing the signer's certificate.
- Fix this issue on the Git client.
- The following step will fail, but if you copy the signer's certificate to the GitLab server, you can use that to do some testing using -nointern -certfile signerscertificate.pem.
Partially verify the commit, using the extracted certificate:
```
/opt/gitlab/embedded/bin/openssl smime -verify -binary -inform pem \
    -in /tmp/signature_text.pk7.pem -content /tmp/signed_text.txt \
    -noverify -certfile /tmp/signature_cert.pem -nointern
```
The output usually includes:
- The parent commit
- The name, email, and timestamp from the commit
- The commit text
- Verification successful (or similar)
This check is not the same as the check GitLab performs, because:
- It does not verify the signer's certificate (-noverify)
- The verification is done using the supplied -certfile rather than the one in the message (-nointern)
Partially verify the commit using the certificate in the message:
```
/opt/gitlab/embedded/bin/openssl smime -verify -binary -inform pem \
    -in /tmp/signature_text.pk7.pem -content /tmp/signed_text.txt \
    -noverify
```
This should get the same result as the previous step, using the extracted certificate.

If the message is missing the certificate, the error includes signer certificate not found.
Fully verify the commit:
```
/opt/gitlab/embedded/bin/openssl smime -verify -binary -inform pem \
    -in /tmp/signature_text.pk7.pem -content /tmp/signed_text.txt
```
If this step fails, verification also fails in GitLab.

Resolve any errors, for example:
- certificate verify error .. unable to get local issuer certificate:
  - The trust chain couldn't be established.
  - This OpenSSL binary uses the GitLab trust store. Either the root certificate is missing from the trust store or the signature is missing the intermediate certificates and a chain to a trusted root can't be built.
    - Intermediate certificates can be put in the trust store if it's not possible to include them in the signature.
    - The procedure for adding certificates to the trust store for packaged GitLab - using /etc/gitlab/trusted-certs.
  - Test additional trusted certificates using OpenSSL with: -CAfile /path/to/rootcertificate.pem
- unsupported certificate purpose:
  - The certificate must specify Digital Signature in the X509v3 Key Usage section of the signer's certificate.
  - If the X509v3 Extended Key Usage (EKU) section is specified, it must include emailProtection. See RFC 5280 for more details:
    
    If there is no purpose consistent with both (Key Usage) extensions, then the certificate MUST NOT be used for any purpose.
    
    If this addition to the EKU list doesn't resolve the issue, please provide feedback in issue 440189.
- signer certificate not found, either:
  - You have added the -nointern argument, but not supplied -certfile.
  - The signature is missing the signer's certificate.