How to pass secrets to Terraform from Google Cloud Build

4 min readSep 27, 2021

Often you need to configure your infrastructure using sensitive or secret information such as usernames, passwords, or API tokens. Do not store sensitive values as plain text in git anymore and use tools that have been built for this purpose and are managed on your behalf. Easily, securely and in a flexible way that supports encryption & versioning.

In the previous article, I have shown you how to create a simple Google Cloud Build pipeline to achieve a per branch CI/CD process using Terraform, and today we will extend the pipeline by adding secret management.

The task

The task is simple — create a MySQL database instance along with a user that can access the database and use password login; automate via Cloud Build and Terraform.

Nicely done! Quick and easy. But wait a minute — is it a good idea to hardcode the password in the Terraform code?

Secret management

It’s a chapter of its own and I will not go into the detail of secrets management in Terraform, especially when there’s a very good and comprehensive write-up on this topic. In a nutshell, we have a few options:

Environment Variables
Encrypted Files (e.g., KMS, PGP, SOPS)
Secret Stores (GCP Secret Manager: GCP-managed key/value store.)

From A comprehensive guide to managing secrets in your Terraform code

For our use case, deploying to GCP, GCP Secret Manager sounds like a feasible option.

The Secret Manager

GCP Secret Manager is very intuitive and easy to use, just navigate to Security > Secret Manager. Click on + CREATE SECRET and insert the value that would normally go into .tfvars file.

Create the secret for the database password

For the next step, note down and copy Resource ID.

Using secrets in the pipeline

Let’s take advantage of the pipeline from the previous article and add our secrets! The only sections we need to add are availableSecrets and secretEnv. An excerpt from the docs:

Replace all the placeholders enclosed by <>and voix-la!

Now we have everything that we need to build a pipeline that retrieves the password value from Secret Manager.

Pipeline

Recap — the source pipeline checks for the existence of a directory in a git repository and then runs Terraform accordingly:

Get the Secret and assign it as an environment variable called TF_VARS
Read the secret
Run terraform

Cloud Build YAML pipeline for Terraform with Secrets

Hang on! But how does that work? How does Terraform know where to get the value for our database password?

The magic happens at line 14 of the pipeline. Before we actually run Terraform there is one important step that needs to take precedence before the plan step. We need to somehow pass the variable from Secret Manager to Terraform. The easiest and the most flexible solution I have found so far is to populate a .auto.tfvars file (docs) with the value of our secret, the database password:

echo "$$TF_VARS" > $BRANCH_NAME.auto.tfvars

When referring to secret variables, you need to use double-dollar sign
(e.g. $$USERNAME) instead of a single $.
Note: GCP Build agents are stateless but in stateful environment it would be a good idea to delete the var file :)

If all went well, Google Cloud build should yield

Successful plan by Terraform with Secret from the environment

Wrap up

Working with secrets is as important as the code you write. When the secrets are passed as ordinary variables and stored in the state file, it is insecure and on top of that — far more difficult to manage or rotate.

How do you manage your secrets? Have you ever thought about that? How do you rotate your secrets?

If you enjoyed the article, I’ll be grateful for any 👏 *CLAPS*, comments, and thoughts!

Don’t forget to subscribe to get updates about new articles.