Building a Robust CI Pipeline for Golang With Google Cloud BuildJake NelsonBlockedUnblockFollowFollowingApr 2Photo by Rodion Kutsaev on UnsplashI have aimed this article directly at Go developers that are interested in CI within GCP and as such, I will intentionally skip steps along the way.
I will try however, where possible, to includes links to other tutorials or documentation on the steps excluded for those who would like to build or refresh their skills.
After you’ve gone through this tutorial, you should have the basic structure of a robust and secure CI pipeline for use with many different Go projects.
The software industry as a whole, has adopted build pipelines as a means to deliver reliable, deterministic, and production ready applications.
To achieve these intrinsic goals in this example we will create multiple steps in our build pipeline to assist us, such as editor-config checks, linting, Go tests, and secure code analysis.
This ensures that our resulting artefact has passed every one of our high standards.
When you first log into Google Cloud Platform you already have a project created for you but if you want to make it easy to destroy/clean-up after this at the end, feel free to create a new project now.
What is Cloud Build?Google Cloud Build gives us an environment in which we can run tests, build/compile, or push artefacts/images.
In Google’s own words:Google Cloud Build lets you build software quickly across all languages.
Get complete control over defining custom workflows for building, testing, and deploying across multiple environments such as VMs, serverless, Kubernetes, or Firebase.
It runs a series of steps specified in a cloudbuild.
yaml file using a specific docker image as the environment for each step whilst sharing a common workspace between steps.
Forking and Cloning the Demo RepoSince this article is aimed at Go developers I’ll assume you know how to fork a repo and clone it to your local machine.
Once you have a local copy of the repo, browse the contents and you’ll see a a couple of dotfiles that we’ll reference later along with our application code.
When using Go within Cloud Build we need to specify our default $GOPATH and our application code then lives under that.
In this example src is our $GOPATH.
Creating our First CloudBuild TriggerOpen Cloud Build or login to the GCP Console and search for it.
If it’s your first time using Cloud Build you’ll be prompted to enable the API so go ahead and do that.
It takes a minute to enable the API, once complete, you’ll be able to create a trigger.
Select your source, we’ll use Github.
Authorise Github and choose the repository you want to trigger on (the fork).
In the trigger settings, you can customise what should trigger a build.
For our CI, we want a push to any branch to trigger the pipeline.
We want to test before anything gets merged into master.
Under Build configuration, we want to use a Cloud Build configuration file aka our cloudbuild.
yaml file.
Create the trigger and review it on the next screen.
The Cloud Build Configuration File (cloudbuild.
yaml)If you browse the local repo you’ll see a cloudbuild.
yaml file will already exist and you won’t need to create it.
The cloudbuild.
yaml file provides Google Cloud Build with a set of instructions (steps) to perform in the CI pipeline.
You could have a range of steps from linting, testing, and finally pushing a resulting artefact.
Note: For the purposes of simplification I am using Docker-hub as a source for some these images.
A more secure practice would be to build the images yourself and store them in Google Container Registry for use in your cloudbuild.
yaml files.
steps: # see https://www.
npmjs.
com/package/editorconfig-checker – id: eclint name: "e53e225/editorconfig-checker" – id: go_version name: "gcr.
io/cloud-builders/go" args: ["version"] env: ["GOPATH=.
"] – id: go_linter name: "golangci/golangci-lint" args: ["golangci-lint","run"] – id: go_test name: "gcr.
io/cloud-builders/go" args: ["test","helloworld"] env: ["GOPATH=.
"] – id: go_security name: "securego/gosec" args: ["helloworld"] env: ["GOPATH=.
"] – id: your_step_here name: "gcr.
io/cloud-builders/go" args: ["run","helloworld"] env: ["GOPATH=.
"]Each of the steps you see above run sequentially within Cloud Build and call an action against the docker image specified by the name field.
I’ll go through each of the steps below and explain what they do to support your CI pipeline.
Cloud-BuildersYou may notice that for some of the step names (or source docker images) the path can contain “gcr.
io/cloud-builders/go”, this means that we’re using a cloud-builder image that Google provides us within GCP.
These images can have several advantages such as:Cached credentials with the service for which they are built (e.
g.
gcr.
io/cloud-builders/git can write to Google Source Repositories)Well maintained (e.
g.
cloud-builders/go is usually consistent with the latest release of Go)They are within GCP meaning that the images don’t need to be pulled from a third party like DockerHubThere’s a community library for the services that aren’t maintained by Google.
Eclint StepThe Eclint step validates the .
editorconfig file located in the root of the repo.
Eclint Step in the cloudbuild.
yamlThe .
editorconfig file enforces formatting standards such as whether indents should be tabs or spaces.
Editors like VSCode will read the file and behave according to the formatting defined in this file.
The .
editorconfig fileGo Version Stepgo_version step from cloudbuild.
yamlWhile this step isn’t actually required, I find it useful to output your version of Go within CI.
If your pipeline breaks suddenly or if a new issue comes up, you can check the version mentioned in the logs and see if it’s changed.
This can give you a good starting point to troubleshoot.
Linter Stepgo_linter step from cloudbuild.
yamlGolangCI-Lint is a great tool for picking up issues before they make it into production.
It’s fast as hell, the developers boast 5x the speed of Gometalinter.
I’ve had fantastic results using it and of course the reduced build times are a huge bonus.
Testing Stepgo_test step from cloudbuild.
yamlJust good old fashioned Go tests, make sure you run them!.You could also output your coverage and validate that you’re meeting your code coverage requirements here too but I might cover that in a separate article for the purposes of brevity.
Security Check Stepgo_security step from cloudbuild.
yamlIt’s important to analyse the code for common security issues.
It’s easy to put in place temporary workarounds to solve problems, and that’s fine but we have to make sure those never make it into production.
We can solve this problem by using a security analyser like Gosec.
Gosec is a fantastic tool that will detect failures to comply with security standards and force a Cloud Build failure.
Cloud Build failure due to insecure codeThat’s it!.You’ve just managed to build a solid foundation for whatever your application code may be.
This tutorial uses Go as an example but you can just as easily apply the same principles to other languages.
In following this tutorial you have:Enforced editorconfig standards so that the whole team is on the same page with formattingLinted the application code to ensure it meets our formatting standards and syntactically makes senseYou’ve run any tests included in your applicationYou’ve analysed the code for common security issuesContinuing on your ownReplace the your_step_here in the cloudbuild.
yaml with something that builds or pushes your results.
Here are some ideas to get you started.
Append a Go build step to compile your code and then push it to a Google Storage Bucket.
Deploy a Golang Google Cloud Function.
Build a docker image and push it to the container registry.
Our cloudbuild.
yaml building successfullyIf you’re interested in seeing an article around any of those steps let me know and I’ll write them up for you.
.. More details