Troubleshoot Serverless Workers on GCP Cloud Run

View Markdown

Currently in: Pre-release

Cloud Run support is in Pre-release, and its APIs may change in backwards-incompatible ways. Create a support ticket or contact your account team for access, and sign up for updates to hear when Cloud Run reaches Public Preview.

This page walks through the Serverless Worker scaling flow on GCP Cloud Run and helps you identify where a failure is occurring. For Lambda functions on AWS, see Troubleshoot Serverless Workers on AWS Lambda.

Cloud Run Workers run in a Worker Pool of long-lived instances. Temporal does not invoke your Worker per Task. Instead, Temporal changes how many instances the pool runs. When the flow works correctly, the following sequence happens:

1. You deploy the Worker image to a Cloud Run Worker Pool with an instance count of zero.
2. You create a Worker Deployment Version pointing at that pool. This starts a Worker Controller Instance (WCI) Workflow.
3. An instance starts, the Worker polls the Temporal Service, and the server binds the Worker's Task Queue to the Worker Deployment Version.
4. The WCI monitors that Task Queue. The Matching Service also signals the WCI when a Task arrives with no Worker free to take it.
5. As work arrives, the WCI raises the pool's instance count through the Cloud Run admin API. As work drains, it lowers the count, potentially to zero.

Start by determining whether the pool is running instances at all.

Is the Worker Pool running instances?

Inspect the pool directly. The instance count the WCI has asked Cloud Run for is stored as an annotation on the pool:

gcloud run worker-pools describe <POOL_NAME> \
  --region <REGION> \
  --project <YOUR_GCP_PROJECT> \
  --format=yaml

Three fields under metadata.annotations tell you most of what you need:

Field	What it tells you
run.googleapis.com/manualInstanceCount	The instance count currently requested. 0 means no Worker is running.
run.googleapis.com/scalingMode	Should be manual. The WCI scales the pool by writing its manual instance count.
serving.knative.dev/lastModifier	Who last changed the pool. If Temporal has ever scaled it, this is your invoker service account.

Use lastModifier to choose where to look next. If lastModifier is still the account you deployed with, Temporal has never successfully written to the pool, so work through the Worker Pool is not scaling up. If it is the invoker service account, Temporal is reaching the pool, and the problem is more likely in the Worker itself. Skip to instances are running but Tasks are not completing.

The Worker Pool is not scaling up

Work through the following checks in order.

Validate the connection to the Worker Pool

Start by verifying that Temporal can reach the pool. Go to Workers > Deployments > select your deployment, open the Actions menu on the version, and click Validate Connection.

For Cloud Run, this impersonates the invoker service account and reads the Worker Pool. A successful validation confirms three things at once: the compute configuration names a pool that exists, Temporal can impersonate the invoker, and the invoker can read the pool. No Worker is started as part of validation.

If validation fails, check each part of the compute configuration against the pool:

• Project, region, and pool name. Temporal addresses the pool as projects/<PROJECT>/locations/<REGION>/workerPools/<POOL_NAME>. A wrong region reports the pool as not found, the same as a wrong name.
• Impersonation. The identity Temporal runs as needs roles/iam.serviceAccountTokenCreator on the invoker service account. On Temporal Cloud, the Terraform module grants this. On a self-hosted Service, you grant it to the GCP identity the server runs as. See Self-hosted setup.
• Invoker permissions. The invoker needs run.workerPools.get to read the pool and run.workerPools.update to scale it. roles/run.developer includes both. Validation only exercises the read, so an invoker that can read but not update validates successfully and then fails to scale.

A passing validation does not prove Temporal can scale the pool. If validation succeeds but lastModifier never changes, check the update permission.

Check that the version is set as current

The Worker Deployment Version must be the current version before Tasks route to it. If you created the version through the CLI, you need to set it as current as a separate step.

Verify with temporal worker deployment describe.

Check that the Task Queue is bound

The WCI scales the pool in response to activity on the Task Queues bound to the Worker Deployment Version. If no Task Queue is bound, there is nothing for it to watch.

Check which Task Queues are bound and whether there is a backlog:

temporal worker deployment describe-version \
  --namespace <NAMESPACE> \
  --deployment-name <DEPLOYMENT_NAME> \
  --build-id <BUILD_ID> \
  --report-task-queue-stats

The server creates the binding when a Worker running that Worker Deployment Version connects and polls. If no Task Queues are listed, no Worker has ever polled successfully under this version. Confirm that an instance can start and connect at all by running the image locally against the same Namespace, or by checking the pool's logs from a run where an instance was up.

Check whether the pool is at its maximum

If the pool is running instances but stops growing while a backlog builds, it may be at the scaling algorithm's upper bound. The max_count setting defaults to 30 instances. Raise it in the Scaling and Lifecycle settings on the Worker Deployment Version.

If the count stalls below max_count, check your project's Cloud Run quotas for the region. Cloud Run caps instances and CPU per region, and those limits apply no matter what count the WCI requests.

Instances are running but Tasks are not completing

If the pool is running instances but Workflows are not progressing, the problem is in the Worker's execution.

Check Worker Pool logs

Read the pool's logs and look for errors during Worker startup:

gcloud run worker-pools logs read <POOL_NAME> \
  --region <REGION> \
  --project <YOUR_GCP_PROJECT>

The pool produces no logs while it is scaled to zero, so run this while an instance is up.

Common errors include:

• Connection failures: The Worker cannot reach the Temporal Service. Check the TEMPORAL_ADDRESS and TEMPORAL_NAMESPACE environment variables on the pool. For self-hosted deployments, verify network reachability from Cloud Run to your Temporal frontend.
• Missing secrets: The instance cannot read the Temporal Cloud API key or TLS material from Secret Manager. The runner service account the pool runs as needs roles/secretmanager.secretAccessor on the secret. This is the service account in spec.template.spec.serviceAccountName, not the invoker service account.
• Authentication errors: The API key is invalid, expired, or has no access to the Namespace.

Check that the deployment name and build ID match

If instances start and poll but no Task is ever processed, the deployment name or build ID in the Worker code may not match the Worker Deployment Version.

Both values must match exactly. Compare the values in your Worker code against the output of temporal worker deployment describe. On a mismatch, the Worker polls under a version the WCI does not manage, so its polls never satisfy the Tasks the WCI is scaling for.

Activities are interrupted mid-execution

If Activities fail partway through and retry from the beginning around the time the pool shrinks, scale-in is stopping instances that are still working.

The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing. It does not track whether the instance Cloud Run stops is mid-Activity.

Use Activity Heartbeats so an interrupted Activity resumes from its last recorded progress instead of restarting.

For how scale-in decisions are made, see Serverless Workers on GCP Cloud Run.