CI/CD with Google Cloud Build, Terraform, and Github

And how to automate it to get closer to GitOps

Photo by Danist Soh on Unsplash

Simple CI/CD using a Github repository, Google cloud build, and Terraform triggered by a commit with different environments on a per branch basis.

First of all, let’s presume that you already have some experience with tools like Google cloud build, CircleCI, Github actions, or similar. Google cloud build is a serverless build, test, and deploy platform wich:

• Build software quickly across all programming languages, including Java, Go, Node.js, and more
• Deploy across multiple environments such as VMs, serverless, Kubernetes, or Firebase
• Access cloud-hosted fully managed CI/CD workflows within your private network
• Keep your data-at-rest within a geographical region or specific location with data residency

Cloud Build Serverless CI/CD Platform | Google Cloud

Setup

Before you execute any pipeline, you need to enable Cloud Build API and configure a service account to have permissions to be able to execute Terraform code and thus manage resources in your GCP project.

• Navigate to Cloud Build, enable Cloud Build API
• Navigate to settings, note your Service account email
• Create your project folder structure

.
├── cloudbuild.yaml
└── terraform
    └── environments
        └── master
            └── main.tf

Cloud Build Settings

• Go to IAM > IAM and grant Editor role to your Service account

Pipeline

Create a cloudbuild.yaml in your project with the following contents

Google Cloud Build pipeline for Terraform

• this pipeline will look for Terraform files located in terraform/environments/<branch> that triggered the build
• if the folder exists for the current branch, it will validate, plan and execute terraform
• if not, Terraform execution will stop after planning which can be handy for feature branches.

To validate the pipeline simply run

gcloud builds submit --config=cloudbuild.yaml . --project <your_project>

Terraform

Create Google Cloud database instance

Tip! If you’re working on your local machine, run the following command first so that terraform is able to authenticate against your GCP project.

gcloud auth application-default login

To be sure that all went well, try running something like the following simple validation of the setup

terraform init
terraform validate
terraform plan

If we didn’t make any mistake, the output of the terraform plan should resemble something like this

Output of terraform plan

And once more validate the pipeline in GCP Build

gcloud builds submit --config=cloudbuild.yaml . --project <your_project>

Why does it take so long to upload a directory that contains one .tf file?

If you look closely on the command, it creates a tarball of the current directory including .terraform folder and its contents which is not very good. Simple solution to that might be:

create .gitignore or .gcloudignore file, more on that in the corresponding documentation.

Automation — Github build triggers

This is the point where a Github repository enters the scene. Navigate back to Cloud Build configuration once more and go to Triggers and connect your repository

• Be sure to create a Push trigger on push to your default branch

In the ideal case, you should end up with a configuration, not unlike the one on the following image

The pipeline can be triggered in multiple ways:

• commit triggered pipeline — to a preselected branch or *,
• manually (just click Run),
• via integration / API / PubSub.

Cleanup

• !!! Remove the Editor role for cloud build service account !!!!
• if you don’t and the account is compromised for any reason, the consequences can be far worse than an unexpectedly huge bill from GCP :)
• How do you deploy Terraform using GCP and Cloud Build?
• Some ideas for improvement — secrets management, feature branches and actual deletion of environments that don’t have any branch assigned, logging and alerting, approvals

Cheers!