1. Introduction to SOPS

1.1 What is SOPS?

First, let’s cut to the chase and get straight to the elephant in the room: what’s SOPS?

SOPS, short for Secrets OPeration S, is an open-source text file editor that encrypts/decrypts files automagically.

Emphasis on the text editor, encryption, and automation.

Typically, when you want to encrypt a text file, this is what you do:

1. Use your favorite editor for writing, editing, and manipulating the text data, and save it as a file.
2. Use an encryption/decryption tool to encrypt the whole file.

When you need to read the encrypted file:

1. First, you need to decrypt the file with an encryption/decryption tool.
2. Open the decrypted file (now it’s a regular text file) with a text editor of your choice.

The drawback of this “normal” process is obvious: you need two tools (an editor and an encryption/decryption tool) for one job.

Enter SOPS.

The easiest way to install SOPS is via brew:

brew install sops

For other OS users, refer to the official GitHub repo of SOPS.

Note: you can definitely use other text editors with SOPS. For VSCode, you even have an extension called vscode-sops that will enable you to auto-decrypt/encrypt files within VSCode with a simple project-based configuration. Please bear in mind this extension is still in early development and may contain bugs.

Even if you don’t want to install the extension, you can still use VSCode to edit the files: set your $EDITOR environment variable to code -wait, (in your ~/.bashrc or ~/.zshrc), and VSCode will open the file when calling sops my-file.yaml.

1.2 How Does It Work?

In short, SOPS offers encryption/decryption and text editing in one package, and the workflow is fully automated. This is where it shines:

• When you write a file, SOPS first uses the specified encryption method of your choice to encrypt the file before saving it to the disk. This is done automatically with no human intervention required whatsoever. Of course, if you open the file with another text editor other than SOPS, you can’t read the content since it’s encrypted.
• When you read the encrypted file with SOPS, it first decrypts the file, opens it up, and shows it to you. Although this sounds like a two-step job, SOPS does it automatically, so it looks like just one single step to the end users.

1.3 Highlights: Flexibility

SOPS supports various encryption methods, like:

• GPG (if you don’t know what it is, read on)
• AWS KMS
• GCP KMS
• HashiCorp Vault
• and so on.

This makes managing and editing sensitive files simple and flexible.

If I want to edit a text file using a local GPG key, no problem; if I want to edit an ENV file and want it to be encrypted by HashiCorp Vault, SOPS also got me covered. I don’t have to use one single encryption method for all of your files because SOPS is highly customizable.

OK, enough said; let’s give it a go to have a hands-on feeling about it.

2 SOPS with PGP Keys

2.1 PGP vs GPG

PGP, GPG, confusing… I know, right?

In short, GPG is a CLI tool that implements PGP:

• PGP (Pretty Good Privacy) is an encryption program that provides cryptographic privacy and authentication for data communication. It can be used for signing, encrypting, and decrypting texts, e-mails, files, directories, and whole disk partitions and to increase the security of e-mail communications.
• GPG, on the other hand, stands for GnuPG, which is a free implementation (a command line tool) of the OpenPGP standard, RFC4880. GPG allows you to encrypt and sign your data and communications; it features a versatile key management system and access modules for all kinds of public key directories.

Now that we’ve got the confusing nomenclature out of the way, let’s create a PGP key using the GPG CLI tool; then, we can test how SOPS works.

2.2 Creating PGP Key with GPG

The easiest way to install GnuPG is via brew, the package manager for macOS:

brew install gnupg

If you are using another OS, for example, Linux, you can use the corresponding package manager. Most likely, this would work:

sudo apt-get install gnupg

With GnuPG, Creating a key is as simple as the following (remember to put your name as the value of KEY_NAME):

export KEY_NAME="Tiexin Guo"
export KEY_COMMENT="test key for sops"gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF

Now, if you run gpg --list-keys, you will see the generated key:

$ gpg --list-keys
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u 
/Users/tiexin/.gnupg/pubring.kbx
--------------------------------
pub rsa4096 2022-09-15 [SCEA]
    A2B73FB4DA0891B38EECD35B47991CD146C9C4BC
uid         [ultimate] Tiexin Guo (test key for sops)
sub rsa4096 2022-09-15 [SEA]

In the “pub” part of the output, you can get the GPG key fingerprint. Alternatively, you can run gpg --list-secret-keys "${KEY_NAME}" to get it.

Store the key fingerprint for future reference.

2.3 SOPS Configuration

We need a simple configuration to let SOPS know we will use the previously generated PGP key for encryption/decryption. To do so, create a file named .sops.yaml under your $HOME directory with the following content:

creation_rules:
    - pgp: >-
        A2B73FB4DA0891B38EECD35B47991CD146C9C4BC

Remember to replace the key fingerprint you generated in the previous step.

OK, now we are done.

2.4 SOPS with Text Files

Run the following command to create a new file named a-text-file.txt:

cd
sops a-text-file.txt

Start editing the content (for example, I put “hello world”), then save.

Now, if we try to cat the file, we won’t be able to get the content:

But according to the output, we can conclude:

• The whole file is encrypted (see the data section from the output above).
• The encryption method (GPG, key fingerprint) is stored in the same file, as part of the file.

To read the file:

sops a-text-file.txt

And we got our original content back!

Note that when reading the file, we do not need to specify which encryption/decryption method to use to read the encrypted file since when we created it in the first place, the encryption/decryption method was already stored as part of the file by SOPS.

2.5 SOPS with Structured Data

Encrypting the full content of a text file automatically is a significant advantage of SOPS, but if the file contains structured data, for example:

• YAML
• JSON
• *.ini
• ENV

SOPS can do even better!

Let’s try:

sops a-yaml-file.yaml

And I put the following content:

username: tiexin
password: test

If we open the file with a regular text editor, say, we cat it:

$ cat a-yaml-file.yaml username: ENC[AES256_GCM,data:h/BVd2tf,iv:IjzYAQQErVeWhwIIuMMfq/pjFr9YJVDNSl6ceRPv+6Q=,tag:2+xJOR89rsOMIQMWHNSEqw==,type:str] password: ENC[AES256_GCM,data:zaz1Jw==,iv:VZ81YN4FRQf14g4olKwb6A8W00BIFT/OgWSEqkrO29s=,tag:tRJHVU7ANvGSw0tOjqGKiQ==,type:str] ...(omitted)

It seems that only the “values” are encrypted, not the “keys”!

This is a huge advantage because if we open this file with a regular text editor, we can still get the structure of the data, only that we can’t read the encrypted content.

I think you know why I’m excited about this feature because I just thought of a great application for it: encrypting Kubernetes secrets YAML files! After encryption, we can still clearly see the content and keys, but not the values, making it safe to store the file somewhere and share it with others.

It’s worth mentioning that for YAML files, SOPS also encrypts comments, another great feature: chances are, sensitive value fields might have some sensitive descriptions in the comment sections, too.

OK, so far, we’ve demonstrated the basic features of SOPS with both standard text files and structured files. But we’ve done everything with PGP keys, which probably isn’t too common to use in a production environment. Next, let’s try SOPS with HashiCorp Vault and AWS KMS, two of the most famous secrets/key managers.

3 SOPS with HashiCorp Vault

3.1 A Short Introduction to Vault

If you don’t know HashiCorp Vault, at its core it is a secrets manager, but it can do much more, including encryption management. Typical use cases include secrets management, encryption services, key management, etc. It’s an open-source project you can run yourself in your own infrastructure; HashiCorp also provides a cloud-based service.

3.2 Set up a Local Vault for Testing

For the SOPS tutorial, we will set up a local Vault service for testing, and the quickest way to do so is using Docker:

docker run -d -p8200:8200 vault:1.2.0 server -dev -dev-root-token-id=toor

Please note that this is only to set up a local development Vault server; do not do this in a production environment. For production usage, check out the official docs.

After running Vault locally in a Docker container, run the following commands to validate:

$ export VAULT_ADDR=http://127.0.0.1:8200
$ export VAULT_TOKEN=toor
$ vault status
Key           Value
---           -----
Seal Type     shamir
Initialized   true
Sealed        false
Total Shares  1
Threshold     1
Version       1.2.0
Cluster Name  vault-cluster-618cc902
Cluster ID    e532e461-e8f0-1352-8a41-fc7c11096908
HA Enabled    false

Once Vault is up and running, let’s create a key to be used by SOPS:

$ # enable a transit engine if not already done
$ # It is suggested to create a transit engine specifically for SOPS
$ vault secrets enable -path=sops transit
Success! Enabled the transit secrets engine at: sops/
$ # create a key
$ vault write sops/keys/firstkey type=rsa-4096
Success! Data written to: sops/keys/firstkey

3.3 Using a Key from Vault with SOPS

Next, let’s edit a file using the key we created in Vault:

sops --hc-vault-transit $VAULT_ADDR/v1/sops/keys/firstkey another-yaml-file.yaml

Write some key/value pairs in YAML format, and save.

If we open the file with another text editor (vim, for example), we can see the Vault-related information is also saved as part of the file. So, when we need to read it, we just run:

sops another-yaml-file.yaml

And that’s it: the process is precisely the same as working with a PGP key.

SOPS supports fine-scoped rules in its config file, so it knows what encryption method and which key to use for each file based on these user-defined rules.

4 SOPS with AWS KMS

4.1 A Short Introduction to KMS

AWS KMS (Key Management Service) is a cloud-based service that helps create and control cryptographic keys. Similarly to HashiCorp Vault, it can create, store and use encryption keys; the difference is that HashiCorp Vault is also a secrets manager, while AWS has a separate service (AWS Secrets Manager) for rotating, managing, and serving secrets.

4.2 Create an AWS KMS Key

For this part, we will use the AWS CLI.

If you haven’t installed it yet, please follow the official doc here. After the installation, we need to configure it properly before using it.

Here we assume that your AWS CLI can already work correctly and has the required permission to access AWS KMS.

Creating a key in AWS KMS using the AWS CLI is surprisingly easy:

aws kms create-key \
    --tags TagKey=Purpose,TagValue=Test \
    --description "Test key"

From the output, copy the ARN, and we can save it as an ENV var (replace the ARN with your own value):

export ARN=arn:aws:kms:ap-southeast-1:858104165444:key/620abad4-8043-4166-8712-2b4684378d3a

4.3 Using a Key from AWS KMS with SOPS

Also simple and straightforward:

sops --kms $ARN aws-kms-encryption-example.yaml

Similarly, the encryption method information is saved as part of the file, and reading the file with SOPS doesn’t require specifying the decryption method, just like using it with GPG or Vault.

SOPS supports more encryption methods, like GCP KMS, Azure Key Vault, etc. Discover more from SOPS’s repo!

5 Summary

5.1 Why You Need SOPS

After following this whole tutorial and executing many commands yourself, you must be a bit tired. Now it’s story time to get relaxed:

Years ago, I was working on a large-scale fin-tech-related project, where the center of the whole infrastructure was an OpenShift cluster (think of it as a customized Kubernetes).

In the cluster, we used namespaces to separate different environments; we used the OpenShift cluster as the single source of truth for secrets management.

For example: if we needed to update the value of a secret, we’d do it in the OpenShift console. If we needed to use a secret in Jenkins, we’d add a secret in OpenShift and it would be automatically synchronized as a Jenkins secret. All seemed fine.

If, however, we wanted to share a secret with another team member or migrate the secret into another cluster/environment, it got tricky: we’d have to export the secrets as K8s secret YAML file and then share the file.

This was both a security and productivity problem: what if the YAML got shared with somebody who wasn’t supposed to have access to that secret? What if the secret in the OpenShift cluster changed, but we didn’t know since our YAML files at hand couldn’t possibly sync with the cluster automatically?

In general, using K8s native secrets as the single source of truth isn’t a best practice, but consuming secrets from places outside a K8s environment isn’t simple. Even if the secrets are only used within the cluster, you’d still have to create hardcoded YAML copies: where do you save those YAML files, and how do you share them safely?

This is where SOPS could help: it encrypts your sensitive files with minimum to no overhead, making it easy to store and share secret files.

5.2 Alternatives for Certain Usecases

There are two similar tools that are a lot less powerful, but, depending on your specific use case, can be equally valuable:

1. Git-crypt enables automated encryption and decryption of files in a git repo. Files that you choose to protect are encrypted when committed and decrypted when checked out. git-crypt lets you freely share a repository containing a mix of public and private content. It’s a great tool if you want to share secrets within your team, but it only supports PGP key encryption; you’d have to generate and manage those PGP keys yourself. And, if you only want to do some encryption/decryption automatically without using git, it doesn’t make sense.
2. Ansible Vault encrypts variables and files to protect sensitive content such as passwords or keys rather than leaving it visible as plaintext in Ansible playbooks or roles. It works similarly to SOPS, but apparently, it can only be used within the Ansible context. Of course, you can also use SOPS with Ansible.

SOPS is still king for generic usage, especially if you want different encryption methods for different files.

5.3 What SOPS Is Not

To be clear: SOPS is not a secrets manager. Yes, you can use it as a quick-and-cheap solution to store sensitive data in encrypted files. But, as your use cases expand, you are likely to hit a wall. Having secrets only in file format has its limitations. For example, you’ll likely need API access to your secrets at some point. SOPS is not made for that, and building a custom solution on top of it would be clunky at best.

This is exactly what secrets managers are made for: allow you flexibility in the formats, policies, and access methods for all your secrets (as files, K8s native YAML, from apps, by actual users, etc.). Modern secret managers have many integrations with CLIs, CI/CD systems, orchestration platforms, etc., so chances are, all your needs could be satisfied without encrypting files.

Recently, I discovered and tested a new kid on the block: Doppler, which can sync secrets to a K8s cluster automatically, eliminating the need to store K8s secrets in YAML files. It can even restart a pod when the dependent secret gets updated. Read more about it here if interested.