Terraform Iteration Desperation: Writing a Module for GCP Pub/Sub

Every one of us has this feeling once in a while: you do your research, pick a tool, start to use it and after a while, you reach a limit of the tool. In most cases, you are using it wrong. In a few, the tool is just not good enough. We all would rather believe that the tool sucks and move on, but let’s face it: we are almost always using it wrong.

I had that feeling once we started to work on a terraform Pub/Sub module. It should accept a map with all the setup we use and create resources for us. The main reason why we decided to create a module was to never ever forget about setting internal service account permissions. Those are required once the dead letter queue is deployed. The problem is that if you forget to set up the service account permissions, GCP will not tell you in logs. It will report the issue in the Pub/Sub console. That might work for administrators who set up their Pub/Sub through the console but not for us.

Setting up correct permissions to service accounts using the Pub/Sub was always complicated for us. Mostly due to problems in the FuQu library which creates a subscription to the topic once the subscription is not present. The idea was that each topic has to have a pull-based subscription. The issue got fixed recently. This behaviour actually means that service accounts have to use the Pub/Sub editor role. The principle of least privilege wasn’t really followed.

There were a few more issues. Overall we decided that enough was enough and the module should have taken care of that. Our rule is that the ugly code can be hidden in the modules but infrastructure repositories should look clean. Boy oh boy, was that true for the Pub/Sub module.

The keys in for_each are always strings

Our idea was to have a map, where each key is a topic. If not defined otherwise, a subscription with the same name is to be created and assigned in pull-based configuration. The problem starts once somebody needs more subscriptions to the same topic. For that, we prepared the following configuration:

"topic-f" : {
  custom_subscriptions : {
    f-sub1 : {
    },
    f-sub2 : {
    }
  }
}

Normally, the code would iterate over a topic and have an inner loop for subscriptions, but! What should be the key to refer to the subscription? Concatenating the topic name and subscription might work. To further obfuscate the setup, there are topics that do not need a subscription and also a few, which we nicknamed black_hole. Those are subscriptions with very limited retention of the messages. Configuration could be written with the following expression:

 subscriptions = toset(flatten([
    for q in google_pubsub_topic.default :
    [
      for j in keys(lookup(var.topics[q.name], "custom_subscriptions", { "${q.name}" : {} })) :
      "${q.name}␟${j}" if !lookup(var.topics[q.name], "black_hole", false) && !lookup(var.topics[q.name], "no_subscription", false)
    ]
  ]))

Ugly, isn’t it? And it gets worse. Simple concatenation is not enough. Should you be able to divide the name of the topic and subscription? Why? Because you need to map the relationship between topic and subscriptions. How else would you say which subscription relates to which topic? Therefore, each item in the set of subscriptions has to be split again:

resource "google_pubsub_subscription" "default" {
  for_each = toset(local.subscriptions)

  topic = split("␟", each.value)[0]
  name  = split("␟", each.value)[1]

...

And what is that separator you ask? It’s “␟” symbol from UTF. I picked that because you wouldn’t find it on the keyboard, and it’s still displayed correctly on most systems. I was tempted to use any other UTF symbols: like 🐕 or 🐱 but I wasn’t able to find anything reasonable. 💩

Adding users to the custom subscription

After many obfuscated pieces of code, I tried to prepare a user setup. It sounds simple: you assign users to topics and subscriptions just by adding a list to correct places in the map:

resource "google_service_account" "service_a" {
  account_id   = "service-a"
  display_name = "service-a"
}

resource "google_service_account" "service_b" {
  account_id   = "service-b"
  display_name = "service-b"
}

module "pubsub" {
  source  = "../"
  topics = {
    "topic-b" : {
      users : [
        "serviceAccount:${google_service_account.service_a.email}",
      ],
    },
    "topic-f" : {
      custom_subscriptions : {
        f-sub : {
          users : [
            "serviceAccount:${google_service_account.service_b.email}",
          ]
        }
      }
    }
  }
}

The problem is that terraform does not know the email addresses of the service accounts while the resources from the pubsub module are being planned. If you create the service accounts before using the module, the problem will not materialize. In case you don’t, terraform reminds you:

│ Error: Invalid for_each argument
│
│   on .terraform/modules/pubsub/iam.tf line 78, in resource "google_pubsub_subscription_iam_member" "user_subscribers":
│   78:   for_each     = toset(local.subscriptions_users)
│     ├────────────────
│     │ local.subscriptions_users is set of string with 4 elements
│
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many
│ instances will be created. To work around this, use the -target argument to first apply only the resources that the for_each
│ depends on.

I would suggest reordering the error message to first say how to work around the issue. I was studying my code for half an hour to notice that it wasn’t exactly my fault. Also, I do understand why this is happening and why I have to create service accounts first. I am just not sure displayed code helped me understand the issue.

Take a look at the module

As you might have noticed, I am not happy with the module. It helps us with having our infrastructure repositories clean, but adjusting anything inside of the module is just painful. If you have ideas on how to rewrite the points above to something nicer, be my guest. The module is available at GitHub.