Skip to main content

Task Queues

Tasks#

A Task is the context that a Worker needs to progress with a specific Workflow Execution or Activity Execution.

There are two types of Tasks:

Task Queues#

A Task Queue is a first-in, first-out queue that a Worker polls for Tasks.

Each Task Queue is capable of queuing both Activity Tasks and Workflow Tasks.

Task Queue component

Task Queue component

Task Queues are very lightweight components.

  • Task Queues do not require explicit registration but instead are created on demand when a Workflow Execution or Activity spawns or when a Worker Process subscribes to it.
  • There is no limit to the number of Task Queues a Temporal Application can use or a Temporal Cluster can maintain.

Workers poll for Tasks in Task Queues via synchronous RPC. This implementation offers several benefits:

  • Worker Processes do not need to have any open ports, which is more secure.
  • Worker Processes do not need to advertise themselves through DNS or any other network discovery mechanism.
  • When all Worker Processes are down, messages simply persist in a Task Queue, waiting for the Worker Processes to recover.
  • A Worker Processes polls for a message only when it has spare capacity, avoiding overloading itself.
  • In effect, Task Queues enable load balancing across a large number of Worker Processes.
  • Task Queues support server-side throttling, which enables you to limit the Task dispatching rate to the pool of Worker Processes while still supporting Task dispatching at higher rates when spikes happen.
  • Task Queues enable what we call Task Routing, which is the routing of specific Tasks to specific Worker Processes or even a specific process.

All Workers listening to a given Task Queue must have identical registrations of Activities and/or Workflows. The one exception is during a Server upgrade, where it is okay to have registration temporarily misaligned while the binary rolls out.

Sticky Execution#

A Sticky Execution is a when a Worker Entity caches the Workflow Execution Event History and creates a dedicated Task Queue to listen on.

A Sticky Execution occurs after a Worker Entity completes the first Workflow Task in the chain of Workflow Tasks for the Workflow Execution.

The Worker Entity caches the Workflow Execution Event History and begins polling the dedicated Task Queue for Workflow Tasks that contain updates, rather than the entire Event History.

If the Worker Entity does not pick up a Workflow Task from the dedicated Task Queue in an appropriate amount of time, the Cluster will resume Scheduling Workflow Tasks on the original Task Queue. Another Worker Entity can then resume the Workflow Execution, and can set up its own Sticky Execution for future Workflow Tasks.

Sticky Executions are the default behavior of the Temporal Platform.

Task Routing#

Task Routing is when a Task Queue is paired with one or more Worker Processes, primarily for Activity Task Executions.

In some use cases, such as file processing, an Activity Task must be routed to a specific Worker Process. For example, suppose that you have a Workflow with the following three separate Activities:

  • Download a file.
  • Process the file in some way.
  • Upload a file to another location.

The first Activity, to download the file, could occur on any Worker on any host. However, the second and third Activities must be executed by a Worker on the same host where the first Activity downloaded the file.

In a real-life scenario, you might have many Worker Processes scaled over many hosts. You would need to develop your Temporal Application to route Tasks to specific Worker Processes when needed.

Code samples:

Get notified of updates