Using pure Golang for Google cloudFrikkyBlockedUnblockFollowFollowingJun 4For the past half year I’ve been playing around with Google cloud’s options for development and deployment using pure Golang.
I soon realized it’s actually pretty hard to use these API’s, at least as a first time user, and want to share parts of why and with what I’ve been struggling.
I’ll be showing some examples and talk about the annoyances of building with Golang for Cloud run (new), Cloud tasks (new ish) and Datastore.
These issues have been experienced every time I want to try a new GCP API, which when you use any cloud platform in general, is quite often.
My struggles have been due to a lack of examples and godoc understanding, and to make matters worse, the code feels really convoluted.
Before moving on, I expect you to have already set up authentication properly.
To get started, here’s a simple PUT and GET from Datastore, with the bare essentials for both.
Since Datastore is a document storage solution (dictionaries, JSON), we’ll just upload a simple JSON object.
For all the following examples you’ll have to define the projectID variable, which is your GCP project name.
Examples here!package mainimport ( "cloud.
google.
com/go/datastore" "context" "encoding/json" "log")// Create an item we're gonna put in and remove// Uses datastore and json tags to map it directlytype Data struct { Key string `datastore:"key" json:"key"`}// Puts some data from the struct Data into the databasefunc putDatastore(ctx context.
Context, client *datastore.
Client, dbname string, data Data) error { // Make a key to map to datastore datastoreKey := datastore.
NameKey(dbname, data.
Key, nil) // Adds the key described above with the // data from datastoreKey if _, err := client.
Put(ctx, datastoreKey, &data); err != nil { log.
Fatalf("Error adding testdata to %s: %s", dbname, err) return err } return nil}// Gets the data back from the datastorefunc getDatastore(ctx context.
Context, client *datastore.
Client, dbname string, identifier string) (*Data, error) { // Defines the key datastoreKey := datastore.
NameKey(dbname, identifier, nil) // Creates an empty variable of struct Data, which we map the data back to newdata := &Data{} if err := client.
Get(ctx, datastoreKey, newdata); err != nil { return &Data{}, err } return newdata, nil}func main() { // Describe the project projectID := "yourprojectnamehere" dbname := "medium-test" ctx := context.
Background() // Create a client client, err := datastore.
NewClient(ctx, projectID) if err != nil { log.
Fatalf("Failed setting up client") } // Create som json data to map to struct jsondata := `{ "key": "qwertyuiopasdfghjkl" }` // Map the jsondata to the struct Data var structData Data if err := json.
Unmarshal([]byte(jsondata), &structData); err != nil { log.
Fatalf("Failed unmarshalling: %s", err) } // Puts the data described above in the datastore if err := putDatastore(ctx, client, dbname, structData); err != nil { log.
Fatalf("Failed putting in datastore: %s", err) } // Gets the same data back from the datastore returnData, err := getDatastore(ctx, client, dbname, structData.
Key) if err != nil { log.
Fatalf("Failed getting from datastore: %s", err) } // Print with some extra value log.
Printf("%#v", returnData)}As for datastore, it has an ok API and is quite simple to use.
You create a client, make a struct to map the data in, and you’re essentially there.
This sample has most of whatever you’ll need.
Moving on, lets have a look at a little more annoying example: Cloud tasks.
I developed these functions before any examples were available, meaning it might not be 100% accurate with live.
(SEE: apiv2beta3).
I used way more time than I feel I should’ve been understanding this API, but the good part is that it taught me how to read and trace godoc pretty better, which has been handy in general.
The code makes a client, defines a project location (which task name to target), creates a single task, and then counts the amount of tasks.
PS: Tasks are autodeleted after the hook, meaning you might have to stop it to test the iterator.
package mainimport ( cloudtasks "cloud.
google.
com/go/cloudtasks/apiv2beta3" "context" "fmt" "google.
golang.
org/api/iterator" taskspb "google.
golang.
org/genproto/googleapis/cloud/tasks/v2beta3" "log")func createTask(ctx context.
Context, client *cloudtasks.
Client, parent string) { // Define some endpoint you want the data to hit from url := "/api/test" // Nested structs.
Just mapped them like this so it's actually readable var appEngineHttpRequest *taskspb.
AppEngineHttpRequest = &taskspb.
AppEngineHttpRequest{ HttpMethod: taskspb.
HttpMethod_GET, RelativeUri: url, } var appeng *taskspb.
Task_AppEngineHttpRequest = &taskspb.
Task_AppEngineHttpRequest{ AppEngineHttpRequest: appEngineHttpRequest, } var task *taskspb.
Task = &taskspb.
Task{ PayloadType: appeng, } // Structs added into the last struct which creates the task req := &taskspb.
CreateTaskRequest{ Parent: parent, Task: task, } ret, err := client.
CreateTask(ctx, req) if err != nil { log.
Printf("Error creating task: %s", err) return } log.
Printf("%#v", ret)}func listAllTasks(ctx context.
Context, client *cloudtasks.
Client, parent string) { // Makes a struct to map req := &taskspb.
ListTasksRequest{ Parent: parent, } // Returns an iterator over the parent tasks and counts ret := client.
ListTasks(ctx, req) cnt := 0 for { _, err := ret.
Next() if err == iterator.
Done { break } if err != nil { log.
Printf("Error in iterator: %s", err) break } cnt += 1 } log.
Printf("Current amount of tasks: %d", cnt)}func main() { // Define the client ctx := context.
Background() client, err := cloudtasks.
NewClient(ctx) if err != nil { log.
Fatalf("Error creating cloudtask client: %s", err) } // Set the projectId, location and queuename for the specific request projectID := "yourprojectnamehere" location := "europe-west3" queuename := "myqueue" var formattedParent string = fmt.
Sprintf("projects/%s/locations/%s/queues/myqueue", projectID, location, queuename) // Creates a task createTask(ctx, client, formattedParent) listAllTasks(ctx, client, formattedParent)}Now, wasn’t that easy?.Well.
Not really.
This is where the hard_to_grasp internals of the google cloud code structure is really making it a hassle for firsttimers.
An example of something I personally found really stupid would be directed at the four nested structs, that are there simply to define a GET request and an endpoint.
I understand that there is some depth to the appengine integration here, but come on.
Before moving on, let’s add a Docker image to the Container Registry on GCP.
Skip this step if you have a webserver docker image ready which listens based on the enviornment variable “PORT”.
(Yes, I’m aware of the hypocrisy, but I don’t really want to use another day just to push a Docker image.
(I’ll get back to this eventually)git clone https://github.
com/frikky/medium-examplescd medium-examples/gcloud/webhook# Set the "projectname" variable in gcp_run.
shvim gcp_run.
sh # .
# Run the script to build and deploy the webserver.
/gcp_run.
sh# run.
sh can will run the same file locallyAs for the last part, Cloud Run, I was excited to see Jaana’s blogpost after the release a little while back, hoping for some real Golang specific examples.
I was sad to see (like most blog posts out there) the fallback is the use of gcloud CLI, and not just native Go code (I understand this is for a broader audience :)).
I like gcloud as much as the next person, but I personally don’t make my platform integrations in bash.
Anyway, here is a snippet that creates a new Cloud Run service for an already existing docker image.
It also has a function to get a service.
Explanation of how I got here is below the code.
I want to emphasize that way more time than a task like this should’ve.
package mainimport ( "context" "fmt" cloudrun "google.
golang.
org/api/run/v1alpha1" "log")func getAllLocations(projectsLocationsService *cloudrun.
ProjectsLocationsService) ([]string, error) { // List locations // Make a request, then do the request list := projectsLocationsService.
List(fmt.
Sprintf("projects/shuffle-241517")) ret, err := list.
Do() if err != nil { log.
Println(err) return []string{}, err } locationNames := []string{} for _, item := range ret.
Locations { locationNames = append(locationNames, item.
Name) } return locationNames, nil}// https://cloud.
google.
com/run/docs/reference/rest/func main() { ctx := context.
Background() // Defines a the projectname, the servicename to use and an existing image to use projectId := "yourprojectnamehere" imagename := "yourimagenamehere" servicename := "webhook2" // Create a service client like anywhere else apiservice, err := cloudrun.
NewService(ctx) if err != nil { log.
Fatalf("Error creating cloudrun service client: %s", err) } // Gets all available locations projectsLocationsService := cloudrun.
NewProjectsLocationsService(apiservice) allLocations, err := getAllLocations(projectsLocationsService) if err != nil { log.
Fatalf("Error getting locations: %s", err) } // Define the service to deploy // Wtf even is this // Metadata initializers // SOO MANY LAYERS OF BULLSHIT (: tmpservice := &cloudrun.
Service{ ApiVersion: "serving.
knative.
dev/v1alpha1", Kind: "Service", Metadata: &cloudrun.
ObjectMeta{ Name: servicename, Namespace: projectId, }, Spec: &cloudrun.
ServiceSpec{ RunLatest: &cloudrun.
ServiceSpecRunLatest{ Configuration: &cloudrun.
ConfigurationSpec{ RevisionTemplate: &cloudrun.
RevisionTemplate{ Metadata: &cloudrun.
ObjectMeta{ DeletionGracePeriodSeconds: 0, }, Spec: &cloudrun.
RevisionSpec{ Container: &cloudrun.
Container{ Image: imagename, Resources: &cloudrun.
ResourceRequirements{ Limits: map[string]string{"memory": "256Mi"}, }, Stdin: false, StdinOnce: false, Tty: false, }, ContainerConcurrency: 80, TimeoutSeconds: 300, }, }, }, }, }, } //Env: []*cloudrun.
EnvVar{ // &cloudrun.
EnvVar{ // Name: "PORT", // Value: "8080", // }, // }, // Deploy the previously described service to all locations // Locations are the same as "parent" in other API calls, AKA: // projects/{projectname}/locations/{locationName} for _, location := range allLocations { getService(projectsLocationsService, location) createService(projectsLocationsService, location, tmpservice) }}func getService(projectsLocationsService *cloudrun.
ProjectsLocationsService, location string) { projectsLocationsServicesGetCall := projectsLocationsService.
Services.
Get(fmt.
Sprintf("%s/services/webhook", location)) service, err := projectsLocationsServicesGetCall.
Do() if err != nil { log.
Fatalf("Error creating new locationservice: %s", err) } _ = service}func createService(projectsLocationsService *cloudrun.
ProjectsLocationsService, location string, service *cloudrun.
Service) { projectsLocationsServicesCreateCall := projectsLocationsService.
Services.
Create(location, service) service, err := projectsLocationsServicesCreateCall.
Do() log.
Println(service, err) if err != nil { log.
Fatalf("Error creating new locationservice: %s", err) } log.
Printf("%#v", service.
Spec)}Now, that might not seem too bad, but to manage the creation of this monstrosity, I had to make it backwards by creating the Get call first.
I initially started building it backwards from the godocs, looking for references to “locations.
services.
create” found here (the REST api doc is _actually_ pretty neat).
After building the structure and testing the API calls, I had to build the “Service” struct, which seemed doable (it’s large and horribly convoluted) until I found that the API didn’t tell me WHAT was wrong when my APIcalls failed.
This means I have to go on a witch hunt for invalid or missing variables within seven (yes, really) nested structs to find the required or missing variables.
Horrible error codeNotes while trying to understand and build the structureSo to further debug what was required, I had a look at the logging utility in Google cloud, and was happy to find that they have ok audit logging.
The real problem though, is that the audit logs don’t tell you anything either.
My issues and anger kept piling up, and I even went to the point of reverse engineering the frontend API calls, walking through required fields when creating a new cloud run service, but to no avail.
The CREATE function calls when creating in the frontend.
I soon remembered that I totally forgot about a “simple” APIcall that I should’ve thought of earlier.
There exists a GET statement for webhooks as well.
I built the GET statement, and soon developed the following struct based on walking all fields of run.
Service returned from the Get Call.
The finished cloudrun example structAnd finally, after all that struggle, I got the long awaited 200 response.
Here’s how it looks in the GUI (Yes, I know I’m amazing at censoring).
Now, what’s the real issue here?.Is it that I’m too dumb to understand the API, the lack of examples, or that it’s in alpha?.Well.
no, not really.
I think there are multiple reasons to my struggles here, with the main one being that it’s simply too convoluted to understand first hand, which in and of itself is it’s own pitfall.
I haven’t done this in other languages, but the JSON behind all that has to be built in any language, so it depends on the wrappers, but with code generation being a big theme in their libaries, I can’t imagine them being much better.
I find it troubling that I used a full day of work just to understand how a Struct should be built.
So there it is, some fully working examples with some context to them.
One easily usable, one a little more tricky with really weird definitions without good samples, and one stupidly complex because of the layered code.
Here’s the source for all three of them.
frikky/medium-examplesExamples from Medium blogposts.
Contribute to frikky/medium-examples development by creating an account on GitHub.
github.
comAgain, the point here isn’t to shit on Google cloud as I love their services and use them daily, but I would like to share my frustration and learnings for features and services that I feel like are missing required documentation for proper implementations.
I’m part of the problem as well, as don’t usually take the time to do pull requests every time I finish something I haven’t found samples for either.
(They don’t accept entirely new samples in the GoogleCloudPlatform/golang-samples repo anyway)Happy serverless coding :).