Skip to main content

Task Queue performance

View Markdown

This page covers Available Task Queue information, Evaluate Task Queue performance, and Task Queue processing tuning.

Available Task Queue information​

info

The information listed in this section is readable using the DescribeTaskQueueEnhanced method in the Go SDK, with the Temporal CLI task-queue describe command, and using DescribeTaskQueue through RPC.

The Temporal Service reports information separately for each Task Queue type (not aggregated). Use the following Task Queue properties to retrieve and evaluate information about Task Queue health and performance. Available data include:

ApproximateBacklogCount and ApproximateBacklogAge​

ApproximateBacklogCount represents the approximate count of Tasks currently backlogged in this Task Queue. The number may include expired Tasks as well as active Tasks, but it will eventually converge to the correct count over time.

ApproximateBacklogAge returns the approximate age of the oldest Task in the backlog. The age is based on the creation time of the Task at the head of the queue.

You can rely on both these counts when making scaling decisions.

Known accuracy limitations​

These values are approximate. The most common sources:

  • Overcount from invalid or expired Tasks: Tasks belonging to cancelled, terminated, completed, or timed out Workflows and Activities stay in the count until they reach the head of the queue and are processed and discarded. An invalid or expired Task at the head is removed quickly, so it rarely holds up the count for long. A valid Task at the head can stay there longer when there aren't enough Workers to dispatch it. While it sits there, invalid or expired Tasks queued behind it cannot be removed. Invalid and expired Tasks are eventually accounted for, but the count may not return to a fully accurate value because of other sources of discrepancy like infrequent metadata updates and database row expirations.
  • Reset to zero on idle Task Queue unload: If a Task Queue sees no activity for approximately 5 minutes - no Worker polls, no new Tasks added, and no other Task Queue calls (DescribeTaskQueue, UpdateTaskQueueConfig, etc.) - the Temporal Service unloads it from memory. When this happens, ApproximateBacklogCount reports zero until the Task Queue is reloaded by the next Worker poll, new Task, or Task Queue API call. An idle Task Queue with a backlog but no active Workers can therefore temporarily report zero even though there are Tasks waiting to be processed.
  • Sticky queue exclusion: Sticky queues are not included in these values. Because Sticky queue Tasks only remain valid for a few seconds, this inaccuracy diminishes as the backlog grows.

TasksAddRate and TasksDispatchRate​

Reports the approximate Tasks-per-second added to or dispatched from a Task Queue. This rate is averaged over the most recent 30-second time interval. The calculations include Tasks that were added to or dispatched from the backlog as well as Tasks that were immediately dispatched and bypassed the backlog (sync-matched).

The actual Task delivery count may be significantly higher than the number reported by these two values:

  • Eager dispatch refers to a Temporal feature where Activities can be requested by an SDK using one Workflow Task completion response. Tasks using Eager dispatch do not pass through Task Queues.
  • Tasks passed to Sticky Task Queues not included in the returned values for TasksAddRate and TasksDispatchRate.

BacklogIncreaseRate​

Approximates the net Tasks per second added to the backlog, averaged over the most recent 30 seconds. This is calculated as:

TasksAddRate - TasksDispatchRate
  • Positive values of X indicate the backlog is growing by about X Tasks per second.
  • Negative values of X indicate the backlog is shrinking by about X Tasks per second.

While individual add and dispatch rates may be inaccurate due to Eager and Sticky Task Queues, the BacklogIncreaseRate reliably reflects the rate at which the backlog is shrinking or growing for backlogs older than a few seconds.

Evaluate Task Queue performance​

A Task Queue is a lightweight, dynamically allocated queue. Worker Entities poll the queue for Tasks and retrieve Tasks to work on. Tasks are contexts that a Worker progresses using a specific Workflow Execution, Activity Execution, or a Nexus Task Execution. Each Task Queue type offers its Tasks to compatible Workers for Task completion. The Temporal Service dynamically creates different Task Queue types including Activity Task Queues, Workflow Task Queues, and Nexus Task Queues.

With an accurate estimate of backlog Tasks, you can determine the optimal number of Workers to deploy. Balance your Worker count with the number of Tasks to achieve the best performance. This approach minimizes Task backlog saturation and reduces idle Workers.

Task Queue data provide numerical insights into your Task Queue activity and backlog characteristics. Use these numbers to tune your production deployments. Evaluate your Worker loads and assess whether you need to scale up or reduce your Worker deployment.

RATE LIMITS

Visibility API rate limits apply to Task Queue performance data requests.

Query Task Queue info with Temporal CLI​

The Temporal CLI helps you monitor and evaluate Worker performance. Issue the following command to display a list of active Workers that have recently polled a Task Queue:

temporal task-queue describe \
    --task-queue YourTaskQueueName \
    [additional options]

This command retrieves poller information, backlog statistics, and task reachability for Task types (available in Temporal Server v1.25.0, Temporal CLI 1.1 and later).

danger

Task reachability status is experimental. Determining Task reachability incurs a non-trivial computing cost. This feature may significantly change or be removed in a future release.

Query Task Queue info with the Go SDK​

Retrieve Task Queue data using the Go SDK by calling DescribeTaskQueueEnhanced. Specify the Task Queue name and set ReportStats to true, as in the following example:

for _, taskQueueName := range taskQueueNames {
        resp, err := s.client.DescribeTaskQueueEnhanced(ctx, client.DescribeTaskQueueEnhancedOptions{
            TaskQueue:   taskQueueName,
            ReportStats: true,
        })
        if err != nil {
            log.Printf("Error describing task queue %s: %v", taskQueueName, err)
        }

        // Get the backlog count from the enhanced response
        backlogCount += getBacklogCount(resp)
    }

Evaluate Worker availability and capacity issues​

Each Temporal Server records the last time of each poll request. This time is displayed in the temporal task-queue describe output.

  • A LastAccessTime value exceeding one minute may indicate that the Worker fleet is at capacity or that Workers have shut down or been removed.

  • Values under 5 minutes typically suggest the Worker fleet is at capacity. "At capacity" means that all Workflow and Activity slots are full.

  • Values over 5 minutes since the last poll request usually suggest that Workers have shut down or been removed. Workers are removed if 5 minutes have passed since the last poll request.

Manage your Worker fleet​

You can adjust the number of Workers to enhance Workflow Execution performance and manage your fleet size. For instance, a large backlog of Tasks with too few Workers will slow down Workflow Execution completions and decrease processing efficiency. Adding more Workers boosts speeds up completion rates and improves throughput. An empty backlog indicates low Worker utilization, allowing you to reduce your fleet and associated costs.

The values provided by temporal task-queue describe can help you manage your Worker fleet deployment:

  • ApproximateBacklogAge shows how long Tasks have been waiting to be dispatched. If this time grows too long, more Workers can boost Workflow efficiency.

  • Calculate the demand per Worker by dividing the number of backlogged Tasks (ApproximateBacklogCount) by the number of Workers. Determine if your task processing rate is within an acceptable range for your needs using the per-Worker demand (how many Tasks each Worker has yet to process), the backlog consumption rate (TasksDispatchRate, the rate at which Workers are processing Tasks), and the dispatch latency (ApproximateBacklogAge, the time the oldest Task has been waiting to be assigned to a Worker).

  • The backlog increase rate (BacklogIncreaseRate) shows the changing demand on your Workers over time. As this rate increases, you may need to add more Workers until demand and capacity are balanced. As it decreases, you may be able to reduce your Worker fleet.

Task Queue processing tuning​

The following steps limit delays in Task Queue processing due to insufficient or unbalanced Workers. Review these steps if you notice high schedule_to_start metrics.

The steps are arranged in the recommended order of execution.

Hosts and resources provisioning​

If currently provisioned Worker hosts are fully utilized (near full CPU usage, high load average, etc), additional Workers hosts have to be provisioned to increase the capacity of the Workers pool.

It's possible to have too many Workers

Monitor the poll success (poll_success/poll_success_sync) and poll timeout poll_timeouts Server metric counters.

Poll Success Rate = (poll_success + poll_success_sync) / (poll_success + poll_success_sync + poll_timeouts)

Poll Success Rate should be >90% in most cases of systems with a steady load. For high volume and low latency, try to target >95%.

If you see

  1. low Poll Success Rate, and
  2. low schedule_to_start_latency, and
  3. low Worker hosts resource utilization at the same time,

then you might have too many workers, consider sizing down.

Worker Executor Slots sizing​

The main area to focus on when tuning is the number of Worker Executor Slots. Increase the maximum number of working slots by adjusting maxConcurrentWorkflowTaskExecutionSize or maxConcurrentActivityExecutionSize if both of the following conditions are met:

  1. The Worker hosts are underutilized (no bottlenecks on CPU, load average, etc.).
  2. The worker_task_slots_available metric from the corresponding Worker type frequently shows a depleted number of available Worker slots.

Alternatively, consider using a resource-based slot supplier as described here.

Poller count​

Sometimes, it can be appropriate to increase the number of task pollers. This is usually more common in situations where your Workers have somewhat high latency when communicating with the server. You can simply use automated poller tuning to handle this automatically.

Consider manual adjustment if:

  1. The Worker hosts are underutilized, for example, there are no bottlenecks on CPU, load average, etc.
  2. worker_task_slots_available metric from the corresponding Worker type shows that a significant percentage of Worker slots are available on a regular basis.
  3. The schedule_to_start metric is abnormally long.

Then consider increasing the number of pollers by adjusting maxConcurrentWorkflowTaskPollers or maxConcurrentActivityTaskPollers, depending on which type of schedule_to_start metric is elevated.

Rate limiting​

If, after adjusting the poller and executors count as specified earlier, you still observe an elevated schedule_to_start, underutilized Worker hosts, or high worker_task_slots_available, you might want to check the following:

  • If server-side rate limiting per Task Queue is set by WorkerOptions#maxTaskQueueActivitiesPerSecond, remove the limit or adjust the value up. (See Go and Java.)
  • If Worker-side rate limiting per Worker is set by WorkerOptions#maxWorkerActivitiesPerSecond, remove the limit. (See Go, TypeScript, and Java.)