Task Queues and Workers are tightly coupled components.
There are two types of Tasks:
Task Queue component
Task Queues are very lightweight components.
- Task Queues do not require explicit registration but instead are created on demand when a Workflow Execution or Activity spawns or when a Worker Process subscribes to it.
- There is no limit to the number of Task Queues a Temporal Application can use or a Temporal Cluster can maintain.
Workers poll for Tasks in Task Queues via synchronous RPC. This implementation offers several benefits:
- Worker Processes do not need to have any open ports, which is more secure.
- Worker Processes do not need to advertise themselves through DNS or any other network discovery mechanism.
- When all Worker Processes are down, messages simply persist in a Task Queue, waiting for the Worker Processes to recover.
- A Worker Processes polls for a message only when it has spare capacity, avoiding overloading itself.
- In effect, Task Queues enable load balancing across a large number of Worker Processes.
- Task Queues support server-side throttling, which enables you to limit the Task dispatching rate to the pool of Worker Processes while still supporting Task dispatching at higher rates when spikes happen.
- Task Queues enable what we call Task Routing, which is the routing of specific Tasks to specific Worker Processes or even a specific process.
All Workers listening to a given Task Queue must have identical registrations of Activities and/or Workflows. The one exception is during a Server upgrade, where it is okay to have registration temporarily misaligned while the binary rolls out.
Task Queues do not have any ordering guarantees. It is possible to have a task that stays in a queue for a long time if there is a backlog that wasn't drained for that time.
A Sticky Execution is a when a Worker Entity caches the Workflow Execution Event History and creates a dedicated Task Queue to listen on.
A Sticky Execution occurs after a Worker Entity completes the first Workflow Task in the chain of Workflow Tasks for the Workflow Execution.
The Worker Entity caches the Workflow Execution Event History and begins polling the dedicated Task Queue for Workflow Tasks that contain updates, rather than the entire Event History.
If the Worker Entity does not pick up a Workflow Task from the dedicated Task Queue in an appropriate amount of time, the Cluster will resume Scheduling Workflow Tasks on the original Task Queue. Another Worker Entity can then resume the Workflow Execution, and can set up its own Sticky Execution for future Workflow Tasks.
Sticky Executions are the default behavior of the Temporal Platform.
Task Routing is when a Task Queue is paired with one or more Worker Processes, primarily for Activity Task Executions.
In some use cases, such as file processing, an Activity Task must be routed to a specific Worker Process. For example, suppose that you have a Workflow with the following three separate Activities:
- Download a file.
- Process the file in some way.
- Upload a file to another location.
The first Activity, to download the file, could occur on any Worker on any host. However, the second and third Activities must be executed by a Worker on the same host where the first Activity downloaded the file.
In a real-life scenario, you might have many Worker Processes scaled over many hosts. You would need to develop your Temporal Application to route Tasks to specific Worker Processes when needed.
- Java file processing example
- PHP file processing example
- Go file processing example - The Go SDK includes a Session feature that abstracts the need to explicitly route Tasks.
In day-to-day conversations, the term Worker is used to denote either a Worker Program, a Worker Process, or a Worker Entity. Temporal documentation aims to be explicit and differentiate between them.
A Worker Program is the static code that defines the constraints of the Worker Process, developed using the APIs of a Temporal SDK.
Component diagram of a Worker Process and the Temporal Server
More formally, a Worker Process is any process that implements the Task Queue Protocol and the Task Execution Protocol.
- A Worker Process is a Workflow Worker Process if the process implements the Workflow Task Queue Protocol and executes the Workflow Task Execution Protocol to make progress on a Workflow Execution. A Workflow Worker Process can listen on an arbitrary number of Workflow Task Queues and can execute an arbitrary number of Workflow Tasks.
- A Worker Process is an Activity Worker Process if the process implements the Activity Task Queue Protocol and executes the Activity Task Processing Protocol to make progress on an Activity Execution. An Activity Worker Process can listen on an arbitrary number of Activity Task Queues and can execute an arbitrary number of Activity Tasks.
Temporal Application developers are responsible for developing Worker Programs and operating Worker Processes. Worker Processes are external to a Temporal Cluster. In many of our tutorials, we show you how to run both a Temporal Cluster and one Worker on the same machine for local development. However, a production-grade Temporal Application typically has a fleet of Worker Processes, all running on hosts external to the Temporal Cluster. A Temporal Application can have as many Worker Processes as needed.
A Worker Process can be both a Workflow Worker Process and an Activity Worker Process. Many SDKs support the ability to have multiple Worker Entities in a single Worker Process. (Worker entity creation and management differ between SDKs.) A single Worker Entity can listen to only a single Task Queue. But if a Worker Process has multiple Worker Entities, the Worker Process could be listening to multiple Task Queues.
Entity relationship diagram (meta model) of Worker Processes, Task Queues, and Tasks
Worker Processes executing Activity Tasks must have access to any resources needed to execute the actions that are defined in Activity Definitions, such as the following:
- Network access for external API calls.
- Credentials for infrastructure provisioning.
- Specialized GPUs for machine learning utilities.
A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.
A Worker Entity listens and polls on a single Task Queue. A Worker Entity contains both a Workflow Worker and an Activity Worker so that it may make progress of either a Workflow Execution or an Activity Execution.