Skip to main content

Temporal Task Queues and Workers

Task Queues and Workers are tightly coupled components.

Tasks

A Task is the context that a Worker needs to progress with a specific Workflow Execution or Activity Execution.

There are two types of Tasks:

Task Queues

A Task Queue is lightweight, dynamically allocated queue that one or more Workers poll for Tasks.

There are two types of Task Queues, Activity Task Queues and Workflow Task Queues. But one of each can exist with the same Task Queue name.

Task Queue component

Task Queue component

Task Queues are very lightweight components.

  • Task Queues do not require explicit registration but instead are created on demand when a Workflow Execution or Activity spawns or when a Worker Process subscribes to it.
  • There is no limit to the number of Task Queues a Temporal Application can use or a Temporal Cluster can maintain.

Workers poll for Tasks in Task Queues via synchronous RPC. This implementation offers several benefits:

  • Worker Processes do not need to have any open ports, which is more secure.
  • Worker Processes do not need to advertise themselves through DNS or any other network discovery mechanism.
  • When all Worker Processes are down, messages simply persist in a Task Queue, waiting for the Worker Processes to recover.
  • A Worker Processes polls for a message only when it has spare capacity, avoiding overloading itself.
  • In effect, Task Queues enable load balancing across a large number of Worker Processes.
  • Task Queues support server-side throttling, which enables you to limit the Task dispatching rate to the pool of Worker Processes while still supporting Task dispatching at higher rates when spikes happen.
  • Task Queues enable what we call Task Routing, which is the routing of specific Tasks to specific Worker Processes or even a specific process.

All Workers listening to a given Task Queue must have identical registrations of Activities and/or Workflows. The one exception is during a Server upgrade, where it is okay to have registration temporarily misaligned while the binary rolls out.

note

Task Queues do not have any ordering guarantees. It is possible to have a task that stays in a queue for a long time if there is a backlog that wasn't drained for that time.

Sticky Execution

A Sticky Execution is a when a Worker Entity caches the Workflow Execution Event History and creates a dedicated Task Queue to listen on.

A Sticky Execution occurs after a Worker Entity completes the first Workflow Task in the chain of Workflow Tasks for the Workflow Execution.

The Worker Entity caches the Workflow Execution Event History and begins polling the dedicated Task Queue for Workflow Tasks that contain updates, rather than the entire Event History.

If the Worker Entity does not pick up a Workflow Task from the dedicated Task Queue in an appropriate amount of time, the Cluster will resume Scheduling Workflow Tasks on the original Task Queue. Another Worker Entity can then resume the Workflow Execution, and can set up its own Sticky Execution for future Workflow Tasks.

Sticky Executions are the default behavior of the Temporal Platform.

Task Routing

Task Routing is when a Task Queue is paired with one or more Worker Processes, primarily for Activity Task Executions.

In some use cases, such as file processing, an Activity Task must be routed to a specific Worker Process. For example, suppose that you have a Workflow with the following three separate Activities:

  • Download a file.
  • Process the file in some way.
  • Upload a file to another location.

The first Activity, to download the file, could occur on any Worker on any host. However, the second and third Activities must be executed by a Worker on the same host where the first Activity downloaded the file.

In a real-life scenario, you might have many Worker Processes scaled over many hosts. You would need to develop your Temporal Application to route Tasks to specific Worker Processes when needed.

Code samples:

Workers

In day-to-day conversations, the term Worker is used to denote either a Worker Program, a Worker Process, or a Worker Entity. Temporal documentation aims to be explicit and differentiate between them.

Worker Program

A Worker Program is the static code that defines the constraints of the Worker Process, developed using the APIs of a Temporal SDK.

Worker Process

Component diagram of a Worker Process and the Temporal Server

Component diagram of a Worker Process and the Temporal Server

A Worker Process is responsible for polling a Task Queue, dequeueing a Task, executing your code in response to a Task, and responding to the Temporal Cluster with the results.

More formally, a Worker Process is any process that implements the Task Queue Protocol and the Task Execution Protocol.

Temporal Application developers are responsible for developing Worker Programs and operating Worker Processes. Worker Processes are external to a Temporal Cluster. In many of our tutorials, we show you how to run both a Temporal Cluster and one Worker on the same machine for local development. However, a production-grade Temporal Application typically has a fleet of Worker Processes, all running on hosts external to the Temporal Cluster. A Temporal Application can have as many Worker Processes as needed.

A Worker Process can be both a Workflow Worker Process and an Activity Worker Process. Many SDKs support the ability to have multiple Worker Entities in a single Worker Process. (Worker entity creation and management differ between SDKs.) A single Worker Entity can listen to only a single Task Queue. But if a Worker Process has multiple Worker Entities, the Worker Process could be listening to multiple Task Queues.

Entity relationship diagram (meta model) of Worker Processes, Task Queues, and Tasks

Entity relationship diagram (meta model) of Worker Processes, Task Queues, and Tasks

Worker Processes executing Activity Tasks must have access to any resources needed to execute the actions that are defined in Activity Definitions, such as the following:

  • Network access for external API calls.
  • Credentials for infrastructure provisioning.
  • Specialized GPUs for machine learning utilities.

Worker Entity

A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.

A Worker Entity listens and polls on a single Task Queue. A Worker Entity contains both a Workflow Worker and an Activity Worker so that it may make progress of either a Workflow Execution or an Activity Execution.

Get notified of updates