Workload Identity for GKE and Service account for Cloud Run

In Ackee we flexibly work with (mainly) Node.js in two runtimes: Google Cloud Run and Google Kubernetes Engine. Though it is not possible in every way, we try to keep the same approach for infrastructure-related stuff like secrets handling. For use cases like this, we do like to have tools which can be used cloud wide. One of the main security improvements we've adopted in recent weeks was Workload Identity for Google Kubernetes Engine. That's one of the tools which helps us use service accounts in a more native way.

Our development and CI system needs to run applications in a few different environments:

• Local development – ideally utilizing Google accounts of our developers (gcloud auth login & gcloud auth application-default login)
• Integration tests – we treat this as a special environment, as we run our own Gitlab Runner machines running outside of GCP – it is pretty okay for us to have a dedicated IAM Service Account for this part, as the test environment is isolated from the production environment and all data is fabricated for each run of test and discarded at test completion
• Production deploy – utilize Workload Identity and avoid creating IAM SA keys

Now let’s see how we handle each of these sections so they play best together and also look at some problems we’ve encountered.

Local development 

As the main advantage of workload identity we see the fact that you don’t need to create and rotate GCP AIM service account keys anymore. So, for local development, developers should use their user's accounts via gcloud SDK. In an ideal case, the developer should need two commands: gcloud auth login

Authorizes user for gcloud commands

gcloud auth application-default login

Allows to “catch” all calls utilizing Application Default Credentials API calls and authorize them with the user's local account credentials.

With this setup, the application deployed in the production pod aka “on server” should use application default credentials via workload identity = they don’t need any GCP IAM service account keys to run and developers use application default credentials via gcloud SDK local authorization = they also don’t need any credentials stored locally. Sounds cool and transparent, but…

APIs that cannot be authenticated by user account

Utilizing Firebase Remote Config API in our project generated following error message:

Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the firebaseremoteconfig.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/…

First we tried to listen to what the error told us and tried to set “billing/quota_project setting” with these commands:

gcloud auth application-default set-quota-project development-project  
gcloud config set billing/quota_project development-project

But without success – the error message stayed. So we had to generate a special service account for local development. For secrets, we utilized Hashicorp Vault a lot so we made this:

Terraform part:

resource "google_project_iam_binding" "web_binding" {  
  project = var.project  
  role = "projects/${var.project}/roles/web.access"  
  members = [  
 "serviceAccount:${google_service_account.web.email}",  
 "serviceAccount:${google_service_account.web_local.email}",  
  ]  
}  
  
resource "google_service_account_key" "web_local_sa" {  
  service_account_id = google_service_account.web_local.name  
}  
  
resource "vault_generic_secret" "web_local_sa_key" {  
  path  = "secret/projects/external/project/development/sa-key-web-local"  
  data_json = jsonencode({ GCP_SERVICE_ACCOUNT_KEY = google_service_account_key.web_local_sa.private_key })  
}  

Then developers of the application should do this before starting local dev server:

vault kv get -format json "secret/projects/external/project/development/sa-key-web-local" | jq -r -c '.data.data.GCP_SERVICE_ACCOUNT_KEY' | base64 -d > "local-dev-gcp-sa.json"  
export GOOGLE_APPLICATION_CREDENTIALS=./local-dev-gcp-sa.json  
export GOOGLE_CLOUD_PROJECT=development-project  

The local development SA key should be easily rotated and developers just fetch new credentials before running the local development environment.

Integration tests

As we run integration tests outside of the GCP environment and actually inside Docker containers managed by docker-compose, we wanted the test environment experience to be as similar to the production environment as possible.

In integration test CI pipeline we reuse GCP IAM service account key we have already saved as Vault credential and save it to CI workspace:

jq .GCP_SERVICE_ACCOUNT_KEY $CI_PROJECT_DIR/secrets-test.json -j > gcp_sa_key.json  

Now we bind-mount this file in docker-compose.yml section “volumes”:

- "$CI_PROJECT_DIR/gcp_sa_key.json:/usr/src/app/gcp_sa_key.json:ro"  

And finally export path to credentials in “environment” section in the same config file:

- GOOGLE_APPLICATION_CREDENTIALS=/usr/src/app/gcp_sa_key.json

With this IAM SA key propagation, integration tests are easily converted to Application Default Credentials approach without running in a GCP environment.

Production deployment of Workload Identity

When deploying to production (and also dev/staging, we treat them the same in almost every way, except for, for example, availability) we always create a custom GCP IAM role and assign it to the service account created for the application. That helps us to have our roles as granular as possible.

For GKE we use Workload Identity to securely bind a service account to the Kubernetes pod without using keys and for Cloud Run we use the “--service-account” argument of “gcloud alpha run deploy” command. Under the hood, we think that both do pretty much the same - authorizing GCP API calls without the need of securely storing and rotating any credentials.

What we’ve learned

GCP offers a native and secure way to handle application credentials needed to communicate with GCP APIs - GCP IAM Service accounts in Google’s terminology.

For Google Cloud Run it is called Cloud Run Service identity. For GKE it is called GKE Workload Identity – these principles allow us not to care about secret keys security for these runtime environments anymore. Together these techniques use a principle called Application Default Credentials.

For integration tests, in situations where we can’t mock tested cloud services, and need to access GCP services, we create GCP IAM SA and set it as credentials for Application Default Credentials, so tests should behave the same as the application in the manner of authentication.

For local development, we tried to use user accounts authentication but failed with some APIs (firebaseremoteconfig.googleapis.com for example). Service account must be used, so we are kinda forced to use GCP IAM SA keys for local development, but we’ve achieved what we wanted – using Application Default Credentials in all environments and not storing secret keys in the application.