What Went Wrong?

The issue arose while I was setting up a new non-production stage for my application. I created a fresh Google Cloud Project, established new service accounts, and configured new secrets, only to face failures in my CI/CD jobs on GitHub. The testing phase was failing, which meant the deployment process was halted before it even began.

The failure was traced back to the Rest API's inability to connect to the intended in-memory database. The logs indicated that the in-memory database was created successfully, but there was a concerning error message indicating a closed connection to an IP address associated with MongoDB Atlas.

Note from the author: Do not attempt to connect to the MongoDB instance referenced above; I have since terminated it. 🙂

Upon investigating the IP address, I found it was a Google IP, which led me to initially suspect that my CI/CD environment may have been compromised. However, this seemed improbable since there was no sensitive data to exploit.

Reflecting on recent changes, I recalled that I had configured a new non-production stage and utilized Secret Manager to generate secrets in that environment. My CI/CD system referenced the same non-production environment to initiate emulated services, which raised suspicions.

Could it be possible that the emulated services were accessing actual Google Cloud production services, including the Secret Manager? One of the secrets I created contained the database URL, prompting me to enable debug logging to verify the actual database URL being used during the build process.

To my astonishment, the logs revealed that the database URL was indeed stored in Google Cloud's secrets, indicating that the application was not connecting to the local in-memory database but rather to a MongoDB Atlas instance on Google Cloud. This misconfiguration caused the build to fail since the MongoDB Atlas was set to deny all unknown incoming connections.

The Root Cause

Examining the configuration of one of my cloud functions, I noted that it included settings such as ingressSettings, memory, CPU allocation, and attached secrets. Although I intended for the cloud platform to attach secrets to the function at runtime, discovering that these secrets were accessible during CI/CD builds and in emulated environments was unexpected.

This led me to consider whether the CI/CD pipeline was configured to impersonate my identity, thus granting it access to production configurations, including sensitive secrets. A quick review of Firebase's documentation regarding CI/CD authentication confirmed my suspicion.

Firebase suggested using the command npx firebase login:ci to initiate an OAuth 2.0 refresh token, allowing the CI/CD system to have access to my personal Google account, which possessed full administrative rights over the Google Cloud project. This posed a significant security risk.

It would have been less problematic had I used a dedicated account lacking owner permissions for the Google Cloud project. However, creating multiple Google accounts can be impractical, especially as Google monitors account activity.

Why Opt for a Service Account?

A service account is a unique type of account designed for applications or compute workloads, rather than individual users. Managed through Identity and Access Management (IAM), service accounts allow applications to operate with dedicated permissions that are independent of personal accounts.

Utilizing service accounts enables us to restrict access to APIs, allowing for read-only or limited write capabilities. For instance, we can create a service account that permits building applications but not deploying them, or one that can access a database while others are restricted.

Had I employed a service account initially, I could have avoided the confusion and time spent troubleshooting this issue. While no user data was compromised this time, larger organizations could face severe consequences from such vulnerabilities.

Additional Considerations

The challenges outlined above are not exclusive to Firebase functions. Functions lacking dedicated service accounts often have more permissions than necessary, as default service accounts created by various Google Cloud services typically possess excessive permissions. This could lead to significant security breaches if an attacker exploits a vulnerability in your application to disrupt services or gain unauthorized access.

To mitigate such risks, I will provide code snippets demonstrating how to run a Google Cloud Function with a dedicated service account. Additionally, I will follow up with a tutorial on setting up service accounts with Firebase, as the existing documentation is insufficient.

For NodeJS-based Google Cloud Functions, a standard service account can be defined using the setGlobalOptions function from the firebase-functions SDK:

import { defineString } from 'firebase-functions/params'

import { setGlobalOptions } from 'firebase-functions/v2/options'

const gcpCloudRegionParam = defineString("DSQ_GCP_CLOUD_REGION")

setGlobalOptions({

region: [gcpCloudRegionParam],

serviceAccount: cloud-run-functions@${process.env.GCLOUD_PROJECT}.iam.gserviceaccount.com

})

When using the gcloud CLI for deployments, the service account can be specified for v2 Functions using the --run-service-account command line argument:

gcloud functions deploy api

--entry-point=api

—gen2

--run-service-account=cloud-run-functions@${PROJECT}.iam.gserviceaccount.com

—source=.

--trigger-http

Thank you for reading! If you have feedback or further suggestions, feel free to reach out to me on Twitter @stfsy.

The first video titled "5 uses for Cloud Functions | Get to know Cloud Firestore #12 - YouTube" explores various applications of Cloud Functions, demonstrating how they can streamline workflows and enhance application functionality.

The second video, "Build a Serverless API with Firebase Cloud Functions, TypeScript and Firestore - YouTube," provides a comprehensive tutorial on constructing a serverless API, showcasing best practices and essential techniques.